Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, hongtao duan, Lu Jiang, lulu hu, Bin Yang, Minying Zhang
摘要
While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval containing 1,216 executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates 800k high quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a 47.4% success rate and a 33.8% All-Pass@4 score, indicating that the enhanced long-horizon error recovery ability synergistically boosts robustness and overall performance. We will release our benchmark, dataset, and models to facilitate future research.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了最后一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper17
- GPT-4V(ision) is a Generalist Web Agent, if GroundedBoyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun 等ICML 2024 · 被引用 496 次
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent CollaborationJunyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang 等NeurIPS 2024 · 被引用 245 次
- OpenCUA: Open Foundations for Computer-Use AgentsXinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang 等NeurIPS 2025 · 被引用 151 次
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task SynthesisQiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin 等ACL 2025 · 被引用 114 次
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform DataZhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li 等ICLR 2026 · 被引用 54 次
相关 Paper
- Anchor: Branch-Point Data Generation for GUI AgentsJinbiao Wei, Yilun Zhao, Kangqi Ni, Arman CohanACL 2026 · 被引用 3 次
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic ModelsZhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang 等CVPR 2026 · 被引用 6 次
- Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-CorrectionYuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen 等ACL 2026 · 被引用 7 次
- WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level FilteringYifei He, Pranit Chawla, Yaser Souri, Subhojit Som 等ACL 2026 · 被引用 5 次
- WebSynthesis: World Model-Guided Monte Carlo Tree Search for Efficient WebAgent Trajectory SynthesisYifei Gao, Junhong Ye, Yifan Yang, Jiaqi Wang 等ACL 2026
