UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization
Zhengxi Lu, Fei Tang, Guangyi Liu, Jin Ma, Kaitao Song, Xu Tan, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
摘要
MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UI-Copilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multi-turn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot's strong generalization to real-world GUI tasks.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了最后一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
引用它的顶会 Paper1
问问它们各自怎么用它它引用的顶会 Paper10
- ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem SolvingZhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen 等ICLR 2024 · 被引用 289 次
- SeeClick: Harnessing GUI Grounding for Advanced Visual GUI AgentsKanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu 等ACL 2024 · 被引用 33 次
- HiconAgent: History Context-aware Policy Optimization for GUI AgentsXurui Zhou, Gongwei Chen, Yuquan Xie, Zaijing Li 等CVPR 2026 · 被引用 11 次
- Test-Time Reinforcement Learning for GUI Grounding via Region ConsistencyYong Du, Yuchen Yan, Fei Tang, Zhengxi Lu 等AAAI 2026 · 被引用 10 次
- SPARK: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic LearningJinyang Wu, Shuo Yang, Yuhao Shen, Shuai Zhang 等ACL 2026 · 被引用 9 次
相关 Paper
- SE-GA: Memory-Augmented Self-Evolution for GUI Agentsshilong jin, Lanjun Wang, Zhuosheng ZhangICML 2026 · 被引用 1 次
- Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model AgentsYi Yu, Liuyi Yao, Yuexiang Xie, Qingquan Tan 等ACL 2026 · 被引用 40 次
- Executable Agentic Memory for GUI AgentZerui Qin, Sheng Yue, Xingyuan Hua, Yongjian Fu 等ICML 2026 · 被引用 1 次
- MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge EvolutionLibo Sun, Jiwen Zhang, Siyuan Wang, Zhongyu WeiACL 2026 · 被引用 5 次
- DeepAgent: A General Reasoning Agent with Scalable ToolsetsXiaoxi Li, Wenxiang Jiao, Jiarui Jin, Guanting Dong 等WWW 2026 · 被引用 38 次
