ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
Jaewook Lee, Filippo Aleotti, Diego Mazala, Guillermo Garcia-Hernando, Sara Vicente, Oliver James Johnston, Isabel Kraus-Liang, Jakub Powierza, Donghoon Shin, Jon E. Froehlich, Gabriel J. Brostow, Jessica Van Brummelen
Abstract
While augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually defined VR environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, the first mobile tool for AI-assisted AR authoring to combine offline scene understanding, fast 3D asset generation, and LLMs -- enabling users to create outdoor scenes through natural language interaction. For example, saying "a dragon enjoying a campfire" (P7) prompts the system to generate and arrange relevant assets, which can then be refined manually. Our technical evaluation shows that our custom pipelines produce more accurate outdoor scene graphs and generate 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI, how users create in freeform use, and design implications for future AR authoring tools. ImaginateAR takes a step toward empowering anyone to create AR experiences anywhere -- simply by speaking their imagination.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Cited by top-tier papers7
- Storycaster: An AI System for Immersive Room-based StorytellingNaisha Agarwal, Judith Amores, Andrew D. WilsonCHI 2026 · 2 citations
- NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision PeopleRuijia Chen, Yuheng Wu, Charlie Houseago, Filipe Gaspar et al.CHI 2026 · 1 citation
- Roomify: Spatially-Grounded Style Transformation for Immersive Virtual EnvironmentsXueyang Wang, Qinxuan Cen, Weitao Bi, Yunxiang Ma et al.CHI 2026 · 1 citation
- SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB SequencesSeok-Young Kim, Dooyoung Kim, Woojin Cho, Hail Song et al.IEEE VR 2026 · 1 citation
- AgentHands: Generating Interactive Hand Gestures for Spatially Grounded Agent Conversations in XRZiyi Liu, David Li, Zhongyi Zhou, David Kim et al.CHI 2026 · 1 citation
Builds on45
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
Related papers
- CoCreatAR: Enhancing Authoring of Outdoor Augmented Reality Experiences Through Asymmetric CollaborationNels Numan, Gabriel J. Brostow, Suhyun Park, Simon Julier et al.CHI 2025 · 12 citations
- DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote EnvironmentsZeyu Wang, Cuong Nguyen, Paul Asente, Julie DorseyCHI 2021 · 47 citations
- SonifyAR: Context-Aware Sound Generation in Augmented RealityXia Su, Jon E. Froehlich, Eunyee Koh, Chang XiaoUIST 2024 · 12 citations
- XRDirector: A Role-Based Collaborative Immersive Authoring SystemMichael Nebeling, Katy Lewis, Yu-Cheng Chang, Lihan Zhu et al.CHI 2020 · 80 citations
- VRCopilot: Authoring 3D Layouts with Generative AI Models in VRLei Zhang, Jin Pan, Jacob Gettig, Steve Oney et al.UIST 2024 · 53 citations
