ACL Findings2026
VANE: Guiding High-Value Exploration in RLVR via Outcome-Process Novelty Shaping
Xu He, Jialiang Guo, Fucheng Xiong, Haodong Zhao, Xingyang Li, Ke Zeng, Xunliang Cai
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) frequently suffers from mode collapse due to the inherent sparsity of feedback signals. While strategies such as entropy regularization introduce randomness, they lack directionality. Simply incorporating diversity rewards is overly one-sided and fails to identify potential logical errors or hallucinations. To address these limitations, we propose VANE (Value-Aligned Novelty Exploration), a method that simultaneously quantifies novelty across the outcome space (via reward or solution divergence) and the semantic process space (via semantic process divergence). Moreover, VANE employs a value-alignment mechanism that symmetrically amplifies scarce, high-quality solutions while explicitly penalizing diverse yet erroneous reasoning paths. Extensive experiments on models such as Qwen2.5-Math-7B across eight benchmarks-encompassing both large-scale mathematical reasoning and out-ofdistribution (OOD) tasks-demonstrate the effectiveness and generalization of the proposed method.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on15
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- Reasoning with Exploration: An Entropy PerspectiveDaixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai et al.AAAI 2026 · 216 citations
- Text2Reward: Reward Shaping with Language Models for Reinforcement LearningTianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu et al.ICLR 2024 · 142 citations
Related papers
- Reinforced Efficient Reasoning via Semantically Diverse ExplorationZiqi Zhao, Zhaochun Ren, Jiahong Zou, Liu Yang et al.ACL 2026 · 5 citations
- Diversity-Incentivized Exploration for Versatile ReasoningZican Hu, Shilin Zhang, Yafu Li, Jianhao Yan et al.ICLR 2026 · 32 citations
- Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement LearningXiaoyun Zhang, Xiaojian Yuan, Di Huang, Wang You et al.ACL Findings 2026
- HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics AlignmentZhanyu Liu, Qingguo Hu, Ante Wang, Chenqing Liu et al.ACL 2026 · 1 citation
- I²B-LPO: Latent Policy Optimization via Iterative Information BottleneckHuilin Deng, Hongchen Luo, Yue Zhu, Long Li et al.ACL 2026
