ACL Findings2025
GUI Agents: A Survey
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi
Abstract
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
34 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
- Reflexion: language agents with verbal reinforcement learning cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities cites Reflexion: language agents with verbal reinforcement learning
- Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Cited by top-tier papers34
- SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from ExperienceZEYI SUN, Ziyu Liu, Yuhang Zang, Yuhang Cao et al.ICML 2026 · 58 citations
- ViMo: A Generative Visual GUI World Model for App AgentsDezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis et al.ICLR 2026 · 19 citations
- MVP: Multiple View Prediction Improves GUI GroundingYunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen et al.CVPR 2026 · 10 citations
- Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and OpportunitiesChangdae Oh, Seongheon Park, To Eun Kim, Jiatong Li et al.ACL 2026 · 8 citations
- BTL-UI: Blink-Think-Link Reasoning Model for GUI AgentShaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang et al.NeurIPS 2025 · 6 citations
Builds on32
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
- VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric TasksWenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu et al.NeurIPS 2023 · 725 citations
Related papers
- UIPro: Unleashing Superior Interaction Capability for GUI AgentsHongxin Li, Jingran Su, Jingfan Chen, Zheng Ju et al.ICCV 2025
- GTA1: GUI Test-time Scaling AgentYan Yang, Dongxu Li, Yutong Dai, Yuhao Yang et al.ICLR 2026 · 109 citations
- MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI AgentsXuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding et al.CVPR 2026
- INREACT: An Inspire-Then-Reinforce Training Framework For Multimodal GUI AgentYuanlei Wang, Liuzhou Zhang, Haohao Luo, Ying ShenEMNLP Findings 2025
- GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI TasksSaelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin et al.CVPR 2026 · 5 citations
