ACL Findings2026
Entropy-Aware Reshaping of Reinforcement Signals for Multi-Answer Reasoning
Abstract
Reinforcement learning with verifiable rewards (RLVR) is a standard post-training paradigm for large language models (LLMs), typically relying on group-wise reward and advantage normalization for stability. In set-valued multianswer tasks, where multiple outputs may be simultaneously correct, this normalization can over-amplify a small number of early highreward samples, suppressing learning signals from other valid generations and leading to overly concentrated updates. We propose Entropy-Aware Reshaping of Reinforcement Signals (EARS), a framework that reshapes how learning signals are normalized and aggregated. EARS uses token-level predictive entropy as an uncertainty cue to compute entropyweighted reward statistics for advantage normalization, encouraging broader exploration and more balanced learning-signal allocation early in training. An adaptive decay schedule then anneals uncertainty-aware reweighting back to standard group normalization to ensure stable convergence. EARS further incorporates a correctness-gated multi-head process reward that provides auxiliary supervision on reasoning traces while remaining aligned with verifiable correctness. Experiments on MC-TACO and MMLU-Multi using Qwen2.5-7B and Llama-3.1-8B-Instruct demonstrate consistent improvements in exact-set accuracy, training stability, and cross-dataset transfer performance on set-valued multi-answer reasoning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Let's Verify Step by Step cites STaR: Bootstrapping Reasoning With Reasoning
- Let's Verify Step by Step cites Training language models to follow instructions with human feedback
Builds on21
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
Related papers
- Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper ReasoningCan Xie, Ruotong Pan, Xiangyu Wu, Yunfei Zhang et al.ACL Findings 2026
- Stable and Efficient Single-Rollout RL for Multimodal ReasoningRui Liu, Dian Yu, Lei Ke, Haolin Liu et al.CVPR 2026 · 13 citations
- Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMsXinzhu Chen, Xuesheng Li, Zhongxiang Sun, Weijie YuACL Findings 2026
- Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning ModelsGuanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian et al.ICLR 2026 · 3 citations
- Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement LearningXiaoyun Zhang, Xiaojian Yuan, Di Huang, Wang You et al.ACL Findings 2026
