GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation
Tao Liu, Chongyu Wang, Rongjie Li, Yingchen Yu, Xuming He, Song Bai
Abstract
While Multimodal Large Language Models (MLLMs) have advanced GUI navigation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present a reasoning-enhanced framework that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of-Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for future steps. Based on this framework, we train a GUI agent, GUI-Rise, through supervised fine-tuning on pseudo-labeled trajectories and reinforcement learning with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including a history-aware objective, directly linking summary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-art results under identical training data conditions, with particularly strong performance in out-of-domain scenarios. These findings validate our framework's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks. Code is available at https://leon022.github.io/GUI-Rise.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext b1d6e2ff-b2a1-46c1-b642-1c312521501fCited by top-tier papers2
- UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy OptimizationZhengxi Lu, Fei Tang, Guangyi Liu, Jin Ma et al.ACL 2026 · 2 citations
- Experience-driven Multi-turn Reinforcement Learning for GUI AgentsZhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen et al.ACL 2026
Builds on12
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- GPT-4V(ision) is a Generalist Web Agent, if GroundedBoyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun et al.ICML 2024 · 496 citations
- AppAgent: Multimodal Agents as Smartphone UsersChi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li et al.CHI 2025 · 57 citations
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI AgentsKanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu et al.ACL 2024 · 33 citations
- MobileGPT: Augmenting LLM with Human-like App Memory for Mobile Task AutomationSunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi et al.MobiCom 2024 · 21 citations
Related papers
- History-Aware Reasoning for GUI AgentsZiwei Wang, Leyang Yang, Xiaoxuan Tang, Sheng Zhou et al.AAAI 2026
- UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement LearningZhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin et al.AAAI 2026 · 103 citations
- Enhancing GUI Agent with Uncertainty-Aware Self-Trained EvaluatorGongwei Chen, Lirong Jie, Lexiao Zou, Weili Guan et al.NeurIPS 2025 · 4 citations
- Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI AutomationYuyang Wanyan, Xi Zhang, Haiyang Xu, Haowei Liu et al.NeurIPS 2025 · 26 citations
- Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM ReasoningZhihao Dou, Qinjian Zhao, Zhongwei Wan, Zhang Dinggen et al.ICML 2026 · 24 citations
