Lune

ACL Findings2025

GUI Agents: A Survey

Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi

Abstract

Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

34 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks cites Deformable DETR: Deformable Transformers for End-to-End Object Detection
  • Reflexion: language agents with verbal reinforcement learning cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
  • Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities cites Reflexion: language agents with verbal reinforcement learning
  • Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers34

  1. SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from ExperienceZEYI SUN, Ziyu Liu, Yuhang Zang, Yuhang Cao et al.ICML 2026 · 58 citations
  2. ViMo: A Generative Visual GUI World Model for App AgentsDezhao Luo, Bohan Tang, Kang Li, Georgios Papoudakis et al.ICLR 2026 · 19 citations
  3. MVP: Multiple View Prediction Improves GUI GroundingYunzhu Zhang, Zeyu Pan, Zhengwen Zeng, Shuheng Shen et al.CVPR 2026 · 10 citations
  4. Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and OpportunitiesChangdae Oh, Seongheon Park, To Eun Kim, Jiatong Li et al.ACL 2026 · 8 citations
  5. BTL-UI: Blink-Think-Link Reasoning Model for GUI AgentShaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang et al.NeurIPS 2025 · 6 citations

Builds on32

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  3. Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
  4. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
  5. VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric TasksWenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu et al.NeurIPS 2023 · 725 citations

Related papers

  1. UIPro: Unleashing Superior Interaction Capability for GUI AgentsHongxin Li, Jingran Su, Jingfan Chen, Zheng Ju et al.ICCV 2025
  2. GTA1: GUI Test-time Scaling AgentYan Yang, Dongxu Li, Yutong Dai, Yuhao Yang et al.ICLR 2026 · 109 citations
  3. MMBench-GUI: A Unified Hierarchical Evaluation Framework for Multi-Platform GUI AgentsXuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding et al.CVPR 2026
  4. INREACT: An Inspire-Then-Reinforce Training Framework For Multimodal GUI AgentYuanlei Wang, Liuzhou Zhang, Haohao Luo, Ying ShenEMNLP Findings 2025
  5. GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI TasksSaelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin et al.CVPR 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines