SHAPE: Stage-aware Hierarchical Advantage via Potential Estimation for LLM Reasoning
Zhengyang Ai, Zikang Shan, Xiaodong Ai, Jingxian Tang, Hangkai Hu, Pinyan Lu
Abstract
Process supervision has emerged as a promising approach for enhancing LLM reasoning, yet existing methods fail to distinguish meaningful progress from mere verbosity, leading to limited reasoning capabilities and unresolved token inefficiency. To address this, we propose Stage-aware Hierarchical Advantage via Potential Estimation (SHAPE), a framework that formalizes reasoning as a trajectory through a state space of empirical solvability. SHAPE introduces a hierarchical credit assignment mechanism: at the segment level, it employs a stage-aware advantage function to prioritize efficient breakthroughs in low-potential states; at the token level, it utilizes entropy-driven redistribution to sharpen execution signals. Extensive experiments in math reasoning across three base models and five benchmarks demonstrate that SHAPE achieves an average accuracy gain of 3% with 30% reduced token consumption.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Cited by top-tier papers1
Ask how each one uses itBuilds on8
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Reasoning with Exploration: An Entropy PerspectiveDaixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai et al.AAAI 2026 · 216 citations
- S-GRPO: Early Exit via Reinforcement Learning in Reasoning ModelsMuzhi Dai, Chenxu Yang, Qingyi SiNeurIPS 2025 · 100 citations
- HybridFlow: A Flexible and Efficient RLHF FrameworkGuangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu et al.EuroSys 2025 · 61 citations
- AttnPO: Attention-Guided Process Supervision for Efficient ReasoningShuaiyi Nie, Siyu Ding, Wenyuan Zhang, Linhao Yu et al.ACL 2026 · 21 citations
Related papers
- Emergent Hierarchical Reasoning in LLMs through Reinforcement LearningHaozhe Wang, Qixin Xu, Che Liu, Junhong Wu et al.ICLR 2026 · 44 citations
- IAPO: Information-Aware Policy Optimization for Token-Efficient ReasoningYinhan He, Yaochen Zhu, Mingjia Shi, Wendy Zheng et al.ICML 2026 · 2 citations
- SSVPO: Effective Step-Level Credit Assignment for RL Training of Language ModelsYugu Li, Zehong Cao, Jianglin Qiao, Siyi HuICLR 2026
- SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMsDachuan Shi, Abedelkadir Asi, Keying Li, Xiangchi Yuan et al.ICLR 2026 · 22 citations
- How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMsZhichen Dong, Yang Li, Yuhan Sun, Weixun Wang et al.ICML 2026 · 1 citation
