RealityCraft: Automated Synthesis of Extended Reality Device Interaction Scripts from Natural Language Instructions

Extended Reality (XR) apps have become increasingly popular across different domains, evidenced by its market growing to over $165 billion in 2024. Despite their widespread usage, XR apps present new challenges for automated software testing approaches, especially for testing script generation. XR apps have complicated action space from multi-device coordination with temporal precision. Besides, natural language instructions in XR apps usually have a large semantic gap with specific XR interactions. Therefore, previous testing script generation approaches on 2D apps such as Android can hardly be adopted into XR apps.