Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, hongtao duan, Lu Jiang, lulu hu, Bin Yang, Minying Zhang
Abstract
While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval containing 1,216 executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates 800k high quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a 47.4% success rate and a 33.8% All-Pass@4 score, indicating that the enhanced long-horizon error recovery ability synergistically boosts robustness and overall performance. We will release our benchmark, dataset, and models to facilitate future research.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 819840bd-bf87-49a8-a19c-e99723b9e280Builds on17
- GPT-4V(ision) is a Generalist Web Agent, if GroundedBoyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun et al.ICML 2024 · 496 citations
- Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent CollaborationJunyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang et al.NeurIPS 2024 · 245 citations
- OpenCUA: Open Foundations for Computer-Use AgentsXinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang et al.NeurIPS 2025 · 151 citations
- OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task SynthesisQiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin et al.ACL 2025 · 114 citations
- ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform DataZhaoyang Liu, Jingjing Xie, Zichen Ding, Zehao Li et al.ICLR 2026 · 54 citations
Related papers
- Anchor: Branch-Point Data Generation for GUI AgentsJinbiao Wei, Yilun Zhao, Kangqi Ni, Arman CohanACL 2026 · 3 citations
- OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic ModelsZhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang et al.CVPR 2026 · 6 citations
- Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-CorrectionYuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen et al.ACL 2026 · 7 citations
- WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level FilteringYifei He, Pranit Chawla, Yaser Souri, Subhojit Som et al.ACL 2026 · 5 citations
- WebSynthesis: World Model-Guided Monte Carlo Tree Search for Efficient WebAgent Trajectory SynthesisYifei Gao, Junhong Ye, Yifan Yang, Jiaqi Wang et al.ACL 2026
