Lune

ACL Findings2026

Policy-Guided Stepwise Action Planning for Controllable LLM Reasoning

Jianpeng Zhou, Qisheng Hu, Jiahai Wang, Wenya Wang

Abstract

Steering large language model (LLM) reasoning via high-level reasoning actions offers a promising approach to improve robustness and interpretability. However, existing action-based paradigms, ranging from training-free prompting to static plan retrieval or prediction, often fail to consistently outperform standard generation because their planners tend to degenerate into repetitive loops or fixed patterns. We propose PG-HAP (Policy-Guided High-Level Action Planning), a lightweight stepwise plannerexecutor framework that learns to select reasoning actions dynamically while keeping the executor LLM fully frozen. The planner is trained with reinforcement learning to optimize answer correctness. To prevent degeneration, we introduce two targeted mechanisms: (i) an Action-Dependency Logit Mask that enforces valid transitions to avoid redundancy, and (ii) an Action Diversity Reward that discourages mode collapse by promoting varied action sequences. Across mathematical and commonsense reasoning benchmarks, PG-HAP improves accuracy over strong baselines while producing less redundant, more adaptive trajectories. This demonstrates that learning high-level planning alone can substantially strengthen reasoning without expensive end-to-end model tuning. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Least-to-Most Prompting Enables Complex Reasoning in Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  4. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  5. Least-to-Most Prompting Enables Complex Reasoning in Large Language ModelsDenny Zhou, Nathanael Schärli, Le Hou, Jason Wei et al.ICLR 2023 · 318 citations

Related papers

  1. Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM ReasoningZhihao Dou, Qinjian Zhao, Zhongwei Wan, Zhang Dinggen et al.ICML 2026 · 24 citations
  2. Learning Planning-based Reasoning by Trajectories Collection and Process Reward SynthesizingFangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen et al.EMNLP 2024 · 1 citation
  3. Policy Guided Tree Search for Enhanced LLM ReasoningYang LiICML 2025
  4. Learning to Plan by Updating Natural LanguageYiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu et al.EMNLP Findings 2024
  5. Teaching LLMs to Plan, Not Just Solve: Plan Learning Boosts LLMs Generalization in Reasoning TasksTianlong Wang, Junzhe Chen, Weibin Liao, Xueting Han et al.EMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines