Lune

ACL Findings2026

Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification

Yiju Guo, Tianyi Hu, Zexu Sun, Yankai Lin

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first low-success prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6× speedup on math reasoning, and a 1.83% gain on scientific and general reasoning. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DAPO: An Open-Source LLM Reinforcement Learning System at Scale cites Scaling Laws for Reward Model Overoptimization
  • Learning to Reason under Off-Policy Guidance cites Scaling Laws for Reward Model Overoptimization
  • Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on17

  1. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  2. Scaling Laws for Reward Model OveroptimizationLeo Gao, John Schulman, Jacob HiltonICML 2023 · 963 citations
  3. Learning to Reason under Off-Policy GuidanceJianhao Yan, Yafu Li, Zican Hu, Zhi Wang et al.NeurIPS 2025 · 310 citations
  4. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMsXumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye et al.ICLR 2026 · 279 citations
  5. DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing ReasoningZhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu et al.ICLR 2026 · 271 citations

Related papers

  1. Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMsXinzhu Chen, Xuesheng Li, Zhongxiang Sun, Weijie YuACL Findings 2026
  2. Contextual Rollout Bandits for Reinforcement Learning with Verifiable RewardsXiaodong Lu, Xiaohan Wang, Jiajun Chai, Guojun Yin et al.ICML 2026 · 7 citations
  3. No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage ShapingThanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Dac Lai et al.ICLR 2026 · 55 citations
  4. Stable and Efficient Single-Rollout RL for Multimodal ReasoningRui Liu, Dian Yu, Lei Ke, Haolin Liu et al.CVPR 2026 · 13 citations
  5. Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy RefinementYunjian Zhang, Sudong Wang, Yang Li, Peiran Xu et al.ICML 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines