Generative Visual Code Mobile World Models
Woosung (Reiss) Koh, Sungjun Han, Segyu Lee, Se-Young Yun, Jay Shin
摘要
Mobile Graphical User Interface (GUI) World Models (WMs) offer a promising path for improving mobile GUI agent performance at trainand inference-time. However, current approaches face a critical trade-off: text-based WMs sacrifice visual fidelity, while the inability of visual WMs in precise text rendering led to their reliance on slow, complex pipelines dependent on numerous external models. We propose a novel paradigm: visual world modeling via renderable code generation, where a single Vision-Language Model (VLM) predicts the next GUI state as executable web code that renders to pixels, rather than generating pixels directly. This combines the strengths of both approaches: VLMs retain their linguistic priors for precise text rendering while their pre-training on structured web code enables high-fidelity visual generation. We introduce gWorld (8B, 32B), the first open-weight visual mobile GUI WMs built on this paradigm, along with a data generation framework that automatically synthesizes code-based training data. In extensive evaluation across 4 in-distribution and 2 out-of-distribution benchmarks, gWorld sets a new pareto frontier in accuracy versus model size, outperforming 8 frontier open-weight models over 50.25× larger. Further analyses show that (1) scaling training data yields meaningful gains, (2) each component of our pipeline improves data quality, and (3) stronger world modeling improves downstream mobile GUI policy performance. Project Page Code gWorld (8B, 32B) MWMBENCH
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper19
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma 等NeurIPS 2022 · 被引用 22,562 次
- Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou 等ICCV 2021 · 被引用 8,921 次
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu 等ICLR 2022 · 被引用 4,966 次
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng 等SOSP 2023 · 被引用 1,016 次
- LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 被引用 865 次
相关 Paper
- AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State MachinesYifan WU, Yiran Peng, Yiyu Chen, Jianhao Ruan 等ICML 2026 · 被引用 16 次
- ViMo: A Generative Visual GUI World Model for App AgentsDezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis 等ICLR 2026 · 被引用 19 次
- GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented UnderstandingDongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang 等ICLR 2025 · 被引用 1 次
- WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement LearningZimu Lu, Houxing Ren, Yunqiao Yang, Ke Wang 等ICLR 2026 · 被引用 12 次
- InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent TrainingZiyun Zhang, Zezhou Wang, Xiaoyi Zhang, Zongyu Guo 等ACL 2026 · 被引用 11 次
