Lune

EMNLP Findings2024

Dynamic Planning for LLM-based Graphical User Interface Automation

Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen, Xinbei Ma, Muyun Yang, Tiejun Zhao, Min Zhang

Abstract

The advent of large language models (LLMs) has spurred considerable interest in advancing autonomous LLMs-based agents, particularly in intriguing applications within smartphone graphical user interfaces (GUIs). When presented with a task goal, these agents typically emulate human actions within a GUI environment until the task is completed. However, a key challenge lies in devising effective plans to guide action prediction in GUI tasks, though planning have been widely recognized as effective for decomposing complex tasks into a series of steps. Specifically, given the dynamic nature of environmental GUIs following action execution, it is crucial to dynamically adapt plans based on environmental feedback and action history. We show that the widely-used ReAct approach fails due to the excessively long historical dialogues. To address this challenge, we propose a novel approach called Dynamic Planning of Thoughts (D-PoT) for LLM-based GUI agents. D-PoT involves the dynamic adjustment of planning based on the environmental feedback and execution history. Experimental results reveal that the proposed D-PoT significantly surpassed the strong GPT-4V baseline by +12.7% (34.66% → 47.36%) in accuracy. The analysis highlights the generality of dynamic planning in different backbone LLMs, as well as the benefits in mitigating hallucinations and adapting to unseen tasks. Code is available at https://github.com/ sqzhang-lazy/D-PoT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices cites OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reflexion: language agents with verbal reinforcement learning cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
  • Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Large Language Models are Zero-Shot Reasoners
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach cites Reflexion: language agents with verbal reinforcement learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile DevicesDezhi Kong, Zhengzhao Feng, Qiliang Liang, Hao Wang et al.CVPR 2026 · 6 citations
  2. Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining ApproachSeoyoung Lee, Seobin Yoon, Seongbeen Lee, Hyesoo Kim et al.UIST 2025 · 2 citations
  3. ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model CapabilitiesYifan Duan, Yihong Tang, Kehai Chen, Liqiang Nie et al.EMNLP 2025
  4. OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser UseXueyu Hu, Tao Xiong, Biao Yi, Zishu Wei et al.ACL 2025

Builds on23

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
  4. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
  5. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li et al.NeurIPS 2023 · 1,778 citations

Related papers

  1. You Only Look at Screens: Multimodal Chain-of-Action AgentsZhuosheng Zhang, Aston ZhangACL Findings 2024 · 28 citations
  2. LLMs Can Plan Only If We Tell ThemBilgehan Sel, Ruoxi Jia, Ming JinICLR 2025
  3. Graph-enhanced Large Language Models in Asynchronous Plan ReasoningFangru Lin, Emanuele La Malfa, Valentin Hofmann, Elle Michelle Yang et al.ICML 2024 · 33 citations
  4. MPO: Boosting LLM Agents with Meta Plan OptimizationWeimin Xiong, Yifan Song, Qingxiu Dong, Bingchan Zhao et al.EMNLP Findings 2025 · 2 citations
  5. AdaPlanner: Adaptive Planning from Feedback with Language ModelsHaotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai et al.NeurIPS 2023 · 257 citations
Dusk over the sea between two cliffs drawn in fine vertical lines