Jackpot: Align Actor-Policy Distribution for scalable and stable RL for LLM
Zhuoming Chen, Hongyi Liu, Yang Zhou, Haizhong Zheng, Beidi Chen
Abstract
Reinforcement learning (RL) has become an increasingly important paradigm for improving large language models (LLMs) on alignment, reasoning, and coding tasks, yet it remains extremely costly. The majority of training time is spent on rollouts. Allowing actor and policy distributions to differ could unlock substantial scalability and efficiency benefits, such as supporting large-batch or asynchronous training, and even enabling a lightweight rollout model. However, existing importance sampling-based corrections for distribution mismatch suffer from an inherent trade-off between stability and training performance. To tackle this problem, we propose Jackpot, which leverages Optimal Budget Rejection Sampling to directly reduce the gap between actor and policy distributions. For efficiency and stability in practical training, We introduce an efficient probability estimation strategy based on Top-K logits with batch bias correction, and designs a stabilized Jackpot-PPO loss that jointly accounts for both the importance sampling ratio and the trust-region constraint in PPO. Empirically, our method achieves stable improvements in large-batch and asynchronous training, and in extreme off-policy training it substantially delays the onset of collapse and delivers competitive performance. Specifically, we achieve 20% improvement on AMC benchmarks and 8% AIME benchmarks over the off-policy baseline under 128× actor-policy update ratio for Qwen3-4B-Base and 64× for Qwen3-8B-Base, while achieving greater stability and better performance than prior off-policy RL methods under extreme settings. More details of the project are available at https://infini-ai-lab.github.io/jpt_website/ . alphabetically ordering based on lastnames A ANALYSIS OF OBRS This appendix provides the theoretical foundation for OBRS. We first formally define the post-rejection distribution that results from our method. We then prove two key results:
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds on8
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi MatiasICML 2023 · 1,472 citations
- SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
- Llemma: An Open Language Model for MathematicsZhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos et al.ICLR 2024 · 433 citations
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language ModelsZiniu Li, Tian Xu, Yushun Zhang, Zhihang Lin et al.ICML 2024 · 165 citations
Related papers
- Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse RolloutsSijia Luo, Xiaokang Zhang, Yuxuan Hu, Bohan Zhang et al.ACL 2026 · 6 citations
- BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive ClippingZhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou et al.ICLR 2026 · 54 citations
- Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-TrainingBrian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain et al.NeurIPS 2025 · 34 citations
- Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment PerspectiveDeyang Kong, Qi Guo, Xiangyu Xi, Wei Wang et al.AAAI 2026 · 5 citations
- Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?Haizhong Zheng, Jiawei Zhao, Beidi ChenICLR 2026 · 57 citations
