Lune

ACL Findings2026

Entropy-Aware Reshaping of Reinforcement Signals for Multi-Answer Reasoning

Zhi Li, Huidan Xu, Zhen Hu, Yali Du, Ying Liu

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a standard post-training paradigm for large language models (LLMs), typically relying on group-wise reward and advantage normalization for stability. In set-valued multianswer tasks, where multiple outputs may be simultaneously correct, this normalization can over-amplify a small number of early highreward samples, suppressing learning signals from other valid generations and leading to overly concentrated updates. We propose Entropy-Aware Reshaping of Reinforcement Signals (EARS), a framework that reshapes how learning signals are normalized and aggregated. EARS uses token-level predictive entropy as an uncertainty cue to compute entropyweighted reward statistics for advantage normalization, encouraging broader exploration and more balanced learning-signal allocation early in training. An adaptive decay schedule then anneals uncertainty-aware reweighting back to standard group normalization to ensure stable convergence. EARS further incorporates a correctness-gated multi-head process reward that provides auxiliary supervision on reasoning traces while remaining aligned with verifiable correctness. Experiments on MC-TACO and MMLU-Multi using Qwen2.5-7B and Llama-3.1-8B-Instruct demonstrate consistent improvements in exact-set accuracy, training stability, and cross-dataset transfer performance on set-valued multi-answer reasoning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Let's Verify Step by Step cites STaR: Bootstrapping Reasoning With Reasoning
  • Let's Verify Step by Step cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on21

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
  5. STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations

Related papers

  1. Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper ReasoningCan Xie, Ruotong Pan, Xiangyu Wu, Yunfei Zhang et al.ACL Findings 2026
  2. Stable and Efficient Single-Rollout RL for Multimodal ReasoningRui Liu, Dian Yu, Lei Ke, Haolin Liu et al.CVPR 2026 · 13 citations
  3. Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMsXinzhu Chen, Xuesheng Li, Zhongxiang Sun, Weijie YuACL Findings 2026
  4. Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning ModelsGuanxu Chen, Yafu Li, Yuxian Jiang, Chen Qian et al.ICLR 2026 · 3 citations
  5. Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement LearningXiaoyun Zhang, Xiaojian Yuan, Di Huang, Wang You et al.ACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines