3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation

Graphical user interface (UI) software has undergone a fundamental transformation from traditional 2D interfaces to spatial 3D environments. While existing work has made remarkable success in 2D software generation, 3D software generation still remains underexplored. Current methods for 3D software generation usually generate 3D environment as a whole and cannot modify specific elements. Furthermore, these methods struggle to handle the complex spatial and semantic constraints inherent in the real world.

To address these challenges, we present Scenethesis, a novel requirement-sensitive 3D software synthesis approach that maintains formal traceability between user specifications and generated 3D software. Scenethesis is built upon ScenethesisLang, a domain-specific language that serves as a granular constraintaware intermediate representation to bridge natural language requirements and 3D software. It serves both as a comprehensive scene description language enabling fine-grained modification of 3D software elements and as a formal constraint-expressive specification language capable of expressing complex spatial constraints. By decomposing 3D software synthesis into stages, Scenethesis enables independent verification, targeted modification, and systematic constraint satisfaction. Our evaluation demonstrates that Scenethesis accurately captures over 80% of user requirements and satisfies more than 90% of hard constraints while handling over 100 constraints simultaneously. Furthermore, Scenethesis achieves a 42.8% improvement in BLIP-2 visual evaluation scores compared to the state-of-the-art method, establishing its effectiveness in generating high-quality 3D software that faithfully adheres to complex user requirements. Source code and supplemental materials are publicly available at https://sites.google.com/view/3d-software-synthesis.