Lune

UIST2024Top-tier venue

VRCopilot: Authoring 3D Layouts with Generative AI Models in VR

Lei Zhang, Jin Pan, Jacob Gettig, Steve Oney, Anhong Guo

Abstract

Immersive authoring provides an intuitive medium for users to create 3D scenes via direct manipulation in Virtual Reality (VR). Recent advances in generative AI have enabled the automatic creation of realistic 3D layouts. However, it is unclear how capabilities of generative AI can be used in immersive authoring to support fluid interactions, user agency, and creativity. We introduce VRCopilot, a mixed-initiative system that integrates pre-trained generative AI models into immersive authoring to facilitate human-AI co-creation in VR. VRCopilot presents multimodal interactions to support rapid prototyping and iterations with AI, and intermediate representations such as wireframes to augment user controllability over the created content. Through a series of user studies, we evaluated the potential and challenges in manual, scaffolded, and automatic creation in immersive authoring. We found that scaffolded creation using wireframes enhanced the user agency compared to automatic creation. We also found that manual creation via multimodal specification offers the highest sense of creativity and agency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

14 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LayoutGPT: Compositional Visual Planning and Generation with Large Language Models cites ATISS: Autoregressive Transformers for Indoor Scene Synthesis
  • ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality cites Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs
  • Zero-1-to-3: Zero-shot One Image to 3D Object cites GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers14

  1. ImaginateAR: AI-Assisted In-Situ Authoring in Augmented RealityJaewook Lee, Filippo Aleotti, Diego Mazala, Guillermo Garcia-Hernando et al.UIST 2025 · 15 citations
  2. InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMsZhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan et al.CHI 2025 · 10 citations
  3. Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and OutputsCyrus Vachha, Yixiao Kang, Zach Dive, Ashwat Chidambaram et al.CHI 2025 · 8 citations
  4. EchoLadder: Progressive AI-Assisted Design of Immersive VR ScenesZhuangze Hou, Jingze Tian, Nianlong Li, Farong Ren et al.UIST 2025 · 6 citations
  5. GaussianNexus: Room-Scale Real-Time AR/VR Telepresence with Gaussian SplattingXincheng Huang, Dieter Frehlich, Ziyi Xia, Peyman Gholami et al.UIST 2025 · 5 citations

Builds on16

  1. Zero-1-to-3: Zero-shot One Image to 3D ObjectRuoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov et al.ICCV 2023 · 1,662 citations
  2. LION: Latent Point Diffusion Models for 3D Shape GenerationXiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic et al.NeurIPS 2022 · 752 citations
  3. GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from ImagesJun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen et al.NeurIPS 2022 · 661 citations
  4. LayoutGPT: Compositional Visual Planning and Generation with Large Language ModelsWeixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani et al.NeurIPS 2023 · 462 citations
  5. ATISS: Autoregressive Transformers for Indoor Scene SynthesisDespoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis et al.NeurIPS 2021 · 293 citations

Related papers

  1. CreepyCoCreator? Investigating AI Representation Modes for 3D Object Co-Creation in Virtual RealityJulian Rasch, Julia Töws, Teresa Hirzle, Florian Müller et al.CHI 2025 · 11 citations
  2. ImmersiProtor: A Collaborative Mixed-Prototype Tool Integrating Spatial Augmented Reality and Component-layered GenerationHongbo Zhang, Yifei Wu, Chaoyi Lin, Weitao You et al.CHI 2026 · 1 citation
  3. Investigating Generative Workflows for Immersive Prototyping: A Study on Scene- and Object-Level Creation in VRCheng-Chih Tsai, Ping-Hsuan Han, Tse-Yu PanIEEE VR 2026
  4. XRDirector: A Role-Based Collaborative Immersive Authoring SystemMichael Nebeling, Katy Lewis, Yu-Cheng Chang, Lihan Zhu et al.CHI 2020 · 80 citations
  5. Compositional Structures as Substrates for Human-AI Co-creation Environment: A Design Approach and A Case StudyYining Cao, Yiyi Huang, Anh Truong, Hijung Valentina Shin et al.CHI 2025 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines