ACL Findings2026
Shorten After You're Right: Lazy Length Penalties for Reasoning RL
Danlong Yuan, Tian Xie, Shaohan Huang, Huishuai Zhang, Zhuocheng Gong, Chong Luo, Furu Wei, Dongyan Zhao
Abstract
Long-reasoning models achieve strong accuracy on complex reasoning tasks, but their extended reasoning trajectories incur substantial memory and latency costs. Several existing shortening methods rely on additional supervision or multi-stage post-training, which primarily reduces inference length and does not reduce the rollout tokens during on-policy reinforcement learning (RL). We instead target onpolicy response shortening, aiming to improve both inference efficiency and RL training throughput. However, because on-policy RL couples optimization with exploration, naively penalizing length can destabilize training and suppress exploration. To impose length pressure safely, we propose a lazy length penalty integrated into the rule-based RL pipeline: it activates only on correct trajectories, only after training accuracy enters a stably improving regime, and only when responses exceed a tolerance band beyond the minimal correct length. Across four settings, our method significantly reduces response length without extra training stages while maintaining or improving performance. In a logic reasoning setting, we achieve a 40% reduction in stepaveraged response length alongside a 14-point gain in performance. For math problems, we reduce step-averaged response length by 33% while preserving performance.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CoT-Valve: Length-Compressible Chain-of-Thought Tuning cites Let's Verify Step by Step
- Dynamic Early Exit in Reasoning Models cites CoT-Valve: Length-Compressible Chain-of-Thought Tuning
- Dynamic Early Exit in Reasoning Models cites Training Language Models to Reason Efficiently
- Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning cites Training Language Models to Reason Efficiently
- Training Language Models to Reason Efficiently cites Let's Verify Step by Step
Cited by top-tier papers1
Builds on15
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- Training Language Models to Reason EfficientlyDaman Arora, Andrea ZanetteNeurIPS 2025 · 270 citations
- Dynamic Early Exit in Reasoning ModelsChenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu et al.ICLR 2026 · 250 citations
- CoT-Valve: Length-Compressible Chain-of-Thought TuningXinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang et al.ACL 2025 · 162 citations
Related papers
- Learning to Reason Efficiently with Discounted Reinforcement LearningAlex Ayoub, Kavosh Asadi, Dale Schuurmans, Csaba Szepesvari et al.ICLR 2026 · 4 citations
- Correct, Concise and Complete: Multi-stage Training For Adaptive ReasoningNathanaël Carraz Rakotonirina, Ren Pang, Neha Anna John, Michael Bohlke-Schneider et al.ACL Findings 2026
- Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning ModelsWei Wu, Liyi Chen, Congxi Xiao, Tianfu Wang et al.ACL 2026 · 1 citation
- Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement LearningHanbing Liu, Lang Cao, Yuanyi Ren, Mengyu Zhou et al.ACL 2026 · 5 citations
- LEASH: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning ModelYanhao Li, Lu Ma, Jiaran Zhang, Lexiang Tang et al.ACL 2026 · 8 citations
