Lune

ACL Findings2026

VANE: Guiding High-Value Exploration in RLVR via Outcome-Process Novelty Shaping

Xu He, Jialiang Guo, Fucheng Xiong, Haodong Zhao, Xingyang Li, Ke Zeng, Xunliang Cai

摘要

Reinforcement Learning with Verifiable Rewards (RLVR) frequently suffers from mode collapse due to the inherent sparsity of feedback signals. While strategies such as entropy regularization introduce randomness, they lack directionality. Simply incorporating diversity rewards is overly one-sided and fails to identify potential logical errors or hallucinations. To address these limitations, we propose VANE (Value-Aligned Novelty Exploration), a method that simultaneously quantifies novelty across the outcome space (via reward or solution divergence) and the semantic process space (via semantic process divergence). Moreover, VANE employs a value-alignment mechanism that symmetrically amplifies scarce, high-quality solutions while explicitly penalizing diverse yet erroneous reasoning paths. Extensive experiments on models such as Qwen2.5-Math-7B across eight benchmarks-encompassing both large-scale mathematical reasoning and out-ofdistribution (OOD) tasks-demonstrate the effectiveness and generalization of the proposed method.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Reasoning with Exploration: An Entropy Perspective 引用了 DAPO: An Open-Source LLM Reinforcement Learning System at Scale
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper15

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes 等ICLR 2020 · 4,112 次引用
  2. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards 等ICLR 2024 · 3,045 次引用
  3. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan 等NeurIPS 2025 · 2,828 次引用
  4. Reasoning with Exploration: An Entropy PerspectiveDaixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai 等AAAI 2026 · 216 次引用
  5. Text2Reward: Reward Shaping with Language Models for Reinforcement LearningTianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu 等ICLR 2024 · 142 次引用

相关 Paper

  1. Reinforced Efficient Reasoning via Semantically Diverse ExplorationZiqi Zhao, Zhaochun Ren, Jiahong Zou, Liu Yang 等ACL 2026 · 5 次引用
  2. Diversity-Incentivized Exploration for Versatile ReasoningZican Hu, Shilin Zhang, Yafu Li, Jianhao Yan 等ICLR 2026 · 32 次引用
  3. Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement LearningXiaoyun Zhang, Xiaojian Yuan, Di Huang, Wang You 等ACL Findings 2026
  4. HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics AlignmentZhanyu Liu, Qingguo Hu, Ante Wang, Chenqing Liu 等ACL 2026 · 1 次引用
  5. I²B-LPO: Latent Policy Optimization via Iterative Information BottleneckHuilin Deng, Hongchen Luo, Yue Zhu, Long Li 等ACL 2026
黄昏的海面,两侧是细线勾勒的悬崖