GenAssist: Interactive Prompt-Driven XR Program Generation

This paper introduces GenAssist, a system for generating interactive Extended Reality (XR) programs from natural language prompts. Given plain text descriptions of desired programs, our system uses Retrieval-Augmented Generation (RAG) to retrieve related documentation and example code, which is then used to prompt Large Language Models (LLMs) to generate and execute hot-pluggable XR programs in real time. To ensure that the programs are written correctly to the user’s specifications, we add a closed-loop feedback mechanism using virtual cameras in the scene that iteratively refines the system’s output, mimicking the development cycle of human developers that compile and then interactively test programs. GenAssist generates scripts that can not only place multiple primitives and 3D models in various locations in a virtual scene, but it can also animate and enable user interactions with those objects. We show that across a benchmark of 50 diverse XR program prompts, our system achieves high output accuracy and program generation quality. Furthermore, we conduct a user study with 18 participants that demonstrates GenAssist’s effectiveness and usability (NASA TLX = 36.6) for XR program generation. We compare GenAssist to prior systems and show that it is significantly faster (<10 seconds per run) and requires fewer LLM calls.