Lune

ACL Findings2026

Entropy Scheduling in Reinforcement Learning for Large Language Models

Xingjin Wang, Howe Tissue, Lu Wang, Linjing Li, Daniel Dajun Zeng

Abstract

We observe that entropy in reinforcement learning functions analogously to the learning rate in LLMs. Maintaining stable entropy, as demonstrated in DAPO (Yu et al., 2025), helps stabilize RL training, while rapid entropy annealing (i.e., so-called entropy collapse) accelerates local performance improvement and enables faster convergence. We argue that these two processes are not antithetical, but can be effectively controlled and scheduled within a single training run, similar to learning rate scheduling. We propose Entropy Schduling (ES), which optimizes different pre-set goals (e.g. k in optimizing Pass@k) by controlling and scheduling entropy at each step of the RL process. We find that maintaining stable entropy early in training followed by entropy annealing achieves superior performance. Moreover, since stable-state entropy and annealed entropy exhibit distinctly different learning dynamics, curriculum learning can be seamlessly integrated to maximize model performance based on different entropy phases. We show that entropy scheduling is straightforward to implement and intuitive in design. Extensive experiments suggest that it delivers consistent and stable performance improvements across diverse models and algorithms.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Reinforcement Learning for Reasoning in Large Language Models with One Training Example cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Reinforcement Learning for Reasoning in Large Language Models with One Training Example cites Let's Verify Step by Step
  • Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning cites Reinforcement Learning for Reasoning in Large Language Models with One Training Example
  • Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning cites Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  2. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  3. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM ReasoningShenzhi Wang, Le Yu, Chang Gao, Chujie Zheng et al.NeurIPS 2025 · 592 citations
  4. Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base ModelJingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang et al.NeurIPS 2025 · 533 citations
  5. Reinforcement Learning for Reasoning in Large Language Models with One Training ExampleYiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren et al.NeurIPS 2025 · 314 citations

Related papers

  1. Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic SchedulingJingchu Gai, Guanning Zeng, Huaqing ZHANG, Han Zhong et al.ICML 2026
  2. Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement LearningTongxi Wang, Zhuoyang Xia, Xinran Chen, Shan LiuICML 2026
  3. Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow OptimizationHuimin Xu, Shuai Zhao, Xiaobao Wu, Anh Tuan LuuACL Findings 2026
  4. Rethinking Entropy Interventions in RLVR: An Entropy Change PerspectiveZhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo et al.ACL 2026 · 42 citations
  5. Targeted Exploration via Unified Entropy Control for Reinforcement LearningChen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao et al.ACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines