Lune

ACL2026Top-tier venue

Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts

Sijia Luo, Xiaokang Zhang, Yuxuan Hu, Bohan Zhang, Ke Wang, Jinbo Su, Mengshu Sun, Lei Liang, Jing Zhang

Abstract

Reinforcement Learning (RL) has become essential for eliciting complex reasoning capabilities in Large Language Models (LLMs). However, the substantial memory overhead of storing Key-Value (KV) caches during longhorizon rollouts acts as a critical bottleneck, often prohibiting efficient training on limited hardware. While existing KV compression techniques offer a remedy for inference, directly applying them to RL training induces a severe policy mismatch, leading to catastrophic performance collapse. To address this, we introduce Sparse-RL, which empowers stable RL training under sparse rollouts. We show that instability arises from a fundamental policy mismatch among the dense old policy, the sparse sampler policy, and the learner policy. To mitigate this issue, Sparse-RL incorporates Sparsity-Aware Rejection Sampling and Importance-based Reweighting to correct the off-policy bias introduced by compressioninduced information loss. Experimental results show that Sparse-RL reduces rollout overhead compared to dense baselines while preserving the performance. Furthermore, Sparse-RL inherently implements sparsity-aware training, significantly enhancing model robustness during sparse inference deployment. The corresponding training data and code are publicly available on the repository 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites Efficient Streaming Language Models with Attention Sinks
  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites SnapKV: LLM Knows What You are Looking for Before Generation
  • KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
  • ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SnapKV: LLM Knows What You are Looking for Before Generation cites Efficient Streaming Language Models with Attention Sinks
  • SnapKV: LLM Knows What You are Looking for Before Generation cites Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. ETS: Energy-Guided Test-Time Scaling for Training-Free RL AlignmentXiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li et al.ICML 2026 · 4 citations
  2. RealChart2Code: Bridging the Gap in Real-World Chart-to-Code Generation via Multi-Task EvaluationJiajun Zhang, Yuying Li, Zhixun Li, Xingyu Guo et al.ACL 2026

Builds on13

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  3. SnapKV: LLM Knows What You are Looking for Before GenerationYuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh et al.NeurIPS 2024 · 1,019 citations
  4. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache QuantizationColeman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney et al.NeurIPS 2024 · 738 citations
  5. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test TimeZichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang et al.NeurIPS 2023 · 557 citations

Related papers

  1. Jackpot: Align Actor-Policy Distribution for scalable and stable RL for LLMZhuoming Chen, Hongyi Liu, Yang Zhou, Haizhong Zheng et al.ICLR 2026
  2. Which Heads Matter for Reasoning? RL-Guided KV Cache CompressionWenjie Du, Li Jiang, Keda TAO, Xue Liu et al.ICML 2026 · 11 citations
  3. Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?Haizhong Zheng, Jiawei Zhao, Beidi ChenICLR 2026 · 57 citations
  4. Sparsity Forcing: Reinforcing Token Sparsity of MLLMsFeng Chen, Yefei He, Lequan Lin, Jing Liu et al.ICLR 2026 · 3 citations
  5. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference RoutersWenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song et al.ICML 2026 · 51 citations
Dusk over the sea between two cliffs drawn in fine vertical lines