Lune

ACL Findings2026

Shorten After You're Right: Lazy Length Penalties for Reasoning RL

Danlong Yuan, Tian Xie, Shaohan Huang, Huishuai Zhang, Zhuocheng Gong, Chong Luo, Furu Wei, Dongyan Zhao

Abstract

Long-reasoning models achieve strong accuracy on complex reasoning tasks, but their extended reasoning trajectories incur substantial memory and latency costs. Several existing shortening methods rely on additional supervision or multi-stage post-training, which primarily reduces inference length and does not reduce the rollout tokens during on-policy reinforcement learning (RL). We instead target onpolicy response shortening, aiming to improve both inference efficiency and RL training throughput. However, because on-policy RL couples optimization with exploration, naively penalizing length can destabilize training and suppress exploration. To impose length pressure safely, we propose a lazy length penalty integrated into the rule-based RL pipeline: it activates only on correct trajectories, only after training accuracy enters a stably improving regime, and only when responses exceed a tolerance band beyond the minimal correct length. Across four settings, our method significantly reduces response length without extra training stages while maintaining or improving performance. In a logic reasoning setting, we achieve a 40% reduction in stepaveraged response length alongside a 14-point gain in performance. For math problems, we reduce step-averaged response length by 33% while preserving performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CoT-Valve: Length-Compressible Chain-of-Thought Tuning cites Let's Verify Step by Step
  • Dynamic Early Exit in Reasoning Models cites CoT-Valve: Length-Compressible Chain-of-Thought Tuning
  • Dynamic Early Exit in Reasoning Models cites Training Language Models to Reason Efficiently
  • Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning cites Training Language Models to Reason Efficiently
  • Training Language Models to Reason Efficiently cites Let's Verify Step by Step
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent PlanningChi-Pin Huang, Yunze Man, Zhiding Yu, Min-Hung Chen et al.CVPR 2026 · 24 citations

Builds on15

  1. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  2. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  3. Training Language Models to Reason EfficientlyDaman Arora, Andrea ZanetteNeurIPS 2025 · 270 citations
  4. Dynamic Early Exit in Reasoning ModelsChenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu et al.ICLR 2026 · 250 citations
  5. CoT-Valve: Length-Compressible Chain-of-Thought TuningXinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang et al.ACL 2025 · 162 citations

Related papers

  1. Learning to Reason Efficiently with Discounted Reinforcement LearningAlex Ayoub, Kavosh Asadi, Dale Schuurmans, Csaba Szepesvari et al.ICLR 2026 · 4 citations
  2. Correct, Concise and Complete: Multi-stage Training For Adaptive ReasoningNathanaël Carraz Rakotonirina, Ren Pang, Neha Anna John, Michael Bohlke-Schneider et al.ACL Findings 2026
  3. Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning ModelsWei Wu, Liyi Chen, Congxi Xiao, Tianfu Wang et al.ACL 2026 · 1 citation
  4. Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement LearningHanbing Liu, Lang Cao, Yuanyi Ren, Mengyu Zhou et al.ACL 2026 · 5 citations
  5. LEASH: Adaptive Length Penalty and Reward Shaping for Efficient Large Reasoning ModelYanhao Li, Lu Ma, Jiaran Zhang, Lexiang Tang et al.ACL 2026 · 8 citations
Dusk over the sea between two cliffs drawn in fine vertical lines