Lune

ACL Findings2026

How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization

Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, Xunliang Cai

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (ii) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradientaware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: https://github.com/GithubX- F/DynaMO-RL.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Reasoning with Exploration: An Entropy Perspective cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Reasoning with Exploration: An Entropy Perspective cites Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
  • Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Does Your Reasoning Model Implicitly Know When to Stop Thinking? cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Does Your Reasoning Model Implicitly Know When to Stop Thinking? cites Real-Time Aligned Reward Model beyond Semantics
  • Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning cites Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  • Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning cites Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
  • Real-Time Aligned Reward Model beyond Semantics cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Does Your Reasoning Model Implicitly Know When to Stop Thinking?Zixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng et al.ICML 2026 · 21 citations
  2. Real-Time Aligned Reward Model beyond SemanticsZixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng et al.ICML 2026 · 18 citations
  3. Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented GenerationPeiyang Liu, Qiang Yan, Ziqiang Cui, Di Liang et al.SIGIR 2026

Builds on13

  1. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  2. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang et al.NeurIPS 2025 · 1,109 citations
  3. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM ReasoningShenzhi Wang, Le Yu, Chang Gao, Chujie Zheng et al.NeurIPS 2025 · 592 citations
  4. Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base ModelJingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang et al.NeurIPS 2025 · 533 citations
  5. Reasoning with Exploration: An Entropy PerspectiveDaixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai et al.AAAI 2026 · 216 citations

Related papers

  1. Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper ReasoningCan Xie, Ruotong Pan, Xiangyu Wu, Yunfei Zhang et al.ACL Findings 2026
  2. Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMsXinzhu Chen, Xuesheng Li, Zhongxiang Sun, Weijie YuACL Findings 2026
  3. ExGRPO: Learning to Reason from ExperienceRunzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu et al.ICLR 2026 · 51 citations
  4. Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVRZijun Min, Bingshuai Liu, Ante Wang, Long Zhang et al.ACL Findings 2026
  5. Rethinking Entropy Interventions in RLVR: An Entropy Change PerspectiveZhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo et al.ACL 2026 · 42 citations
Dusk over the sea between two cliffs drawn in fine vertical lines