D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies
Sen Chen, Tong Zhao, Yi Bin, Fei Ma, Wenqi Shao, Zheng Wang
Abstract
Developing intelligent agents capable of operating a wide range of Graphical User Interfaces (GUIs) with human-level proficiency is a key milestone on the path toward Artificial General Intelligence. While most existing datasets and benchmarks for training and evaluating GUI agents are static and idealized, failing to reflect the complexity and unpredictability of real-world environments, particularly the presence of anomalies. To bridge this research gap, we propose D-GARA, a dynamic benchmarking framework, to evaluate Android GUI agent robustness in real-world anomalies. D-GARA introduces a diverse set of real-world anomalies that GUI agents commonly face in practice, including interruptions such as permission dialogs, battery warnings, and update prompts. Based on D-GARA framework, we construct and annotate a benchmark featuring commonly used Android applications with embedded anomalies to support broader community research. Comprehensive experiments and results demonstrate substantial performance degradation in state-of-the-art GUI agents when exposed to anomaly-rich environments, highlighting the need for robustness-aware learning. D-GARA is modular and extensible, supporting the seamless integration of new tasks, anomaly types, and interaction scenarios to meet specific evaluation goals.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents cites SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents cites SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
- GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents cites ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use cites SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Builds on7
- GUI-Actor: Coordinate-Free Visual Grounding for GUI AgentsQianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang et al.NeurIPS 2025 · 98 citations
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI AgentsKanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu et al.ACL 2024 · 33 citations
- Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language ModelsWenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu et al.EMNLP Findings 2024 · 32 citations
- ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer UseKaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo et al.ACM MM 2025 · 24 citations
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous AgentsChristopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz et al.ICLR 2025
Related papers
- MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World EnvironmentQinzhuo Wu, Zhizhuo Yang, Hanhao Li, Pengzhi Gao et al.ACL Findings 2026
- AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI AgentsYue Cao, Yingyao Wang, Pi Bu, Jingxuan Xing, Wei Jiang, Zekun Zhu, Junpeng Ma, Sashuai Zhou, Tong Lu, Jun Song, Yu Cheng, Yuning Jiang, Bo ZhengCVPR Findings 2026
- ProBench: Benchmarking GUI Agents with Accurate Process InformationLeyang Yang, Ziwei Wang, Xiaoxuan Tang, Sheng Zhou et al.AAAI 2026
- VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability DiagnosticsYichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou et al.ICML 2026 · 1 citation
- Agent-SAMA: State-Aware Mobile AssistantLinqiang Guo, Wei Liu, Yi Wen Heng, Tse-Hsun (Peter) Chen et al.AAAI 2026 · 2 citations
