GUI-G²: Gaussian Reward Modeling for GUI Grounding
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, Yueting Zhuang
摘要
Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction. Current reinforcement learning approaches use binary rewards that treat elements as hit-or-miss targets, creating sparse signals that ignore the continuous nature of spatial interactions. Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards (GUI-G2), a principled reward framework that models GUI elements as continuous Gaussian distributions across the interface plane. GUI-G2 incorporates two synergistic mechanisms: Gaussian point rewards model precise localization through exponentially decaying distributions centered on element centroids, while coverage rewards assess spatial alignment by measuring the overlap between predicted Gaussian distributions and target regions. To handle diverse element scales, we develop an adaptive variance mechanism that calibrates reward distributions based on element dimensions. This framework transforms GUI grounding from sparse binary classification to dense continuous optimization, where Gaussian distributions generate rich gradient signals that guide models toward optimal interaction positions. Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks demonstrate that GUI-G2, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides superior robustness to interface variations and enhanced generalization to unseen layouts, establishing a new paradigm for spatial reasoning in GUI interaction tasks.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了最后一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
引用它的顶会 Paper6
- Careful Queries, Credible Results: Teaching RAG Models Advanced Web Search Tools with Reinforcement LearningYuqin Dai, Shuo Yang, Guoqing Wang, Yong Deng 等AAAI 2026 · 被引用 5 次
- Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference OptimizationXinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen 等CVPR 2026 · 被引用 4 次
- CGL: Advancing Continual GUI Learning via Reinforcement Fine-TuningZhenquan Yao, Zitong Huang, Yihan Zeng, Jianhua Han 等CVPR 2026 · 被引用 4 次
- GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI AgentsYang Li, Yuchen Liu, Haoyu Lu, Zhiqiang Xia 等CVPR 2026 · 被引用 3 次
- Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI AutomationZehao Deng, Tianjie Ju, Zheng Wu, Zhuosheng Zhang 等CVPR 2026 · 被引用 1 次
它引用的顶会 Paper18
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao 等ICCV 2023 · 被引用 13,211 次
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 被引用 2,600 次
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li 等NeurIPS 2023 · 被引用 1,778 次
- Video-R1: Reinforcing Video Reasoning in MLLMsKaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo 等NeurIPS 2025 · 被引用 528 次
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent CollaborationJunyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang 等NeurIPS 2024 · 被引用 245 次
相关 Paper
- Test-Time Reinforcement Learning for GUI Grounding via Region ConsistencyYong Du, Yuchen Yan, Fei Tang, Zhengxi Lu 等AAAI 2026 · 被引用 10 次
- GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI AgentsYuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou 等NeurIPS 2025 · 被引用 73 次
- GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI AgentsChen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu 等AAAI 2026 · 被引用 5 次
- FDC-Ground: Improving GRPO for GUI Grounding via Exponential Rewards and Fact-Aligned PruningXiangjian Zeng, Wenjing Li, Qingqiang Wu, Liang ZhangAAAI 2026 · 被引用 1 次
- Learning GUI Grounding with Spatial Reasoning from Visual FeedbackYu Zhao, Wei-Ning Chen, Huseyin Inan, Samuel Kessler 等ICML 2026 · 被引用 9 次
