REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Drew Bagnell, Jason D. Lee, Wen Sun
Abstract
While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications, including the fine-tuning of generative models. Unfortunately, PPO requires multiple heuristics to enable stable convergence (e.g. value networks, clipping), and is notorious for its sensitivity to the precise implementation of these components. In response, we take a step back and ask what a minimalist RL algorithm for the era of generative models would look like. We propose REBEL, an algorithm that cleanly reduces the problem of policy optimization to regressing the relative reward between two completions to a prompt in terms of the policy, enabling strikingly lightweight implementation. In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL, which allows us to match the strongest known theoretical guarantees in terms of convergence and sample complexity in the RL literature. REBEL can also cleanly incorporate offline data and be extended to handle the intransitive preferences we frequently see in practice. Empirically, we find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO, all while being simpler to implement and more computationally efficient than PPO. When fine-tuning Llama-3-8B-Instruct, REBEL achieves strong performance in AlpacaEval 2.0, MT-Bench, and Open LLM Leaderboard.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
40 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO cites Training language models to follow instructions with human feedback
- What Makes a Reward Model a Good Teacher? An Optimization Perspective cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- What Makes a Reward Model a Good Teacher? An Optimization Perspective cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- What Makes a Reward Model a Good Teacher? An Optimization Perspective cites Training language models to follow instructions with human feedback
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling cites High-Resolution Image Synthesis with Latent Diffusion Models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage cites LoRA: Low-Rank Adaptation of Large Language Models
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- The Importance of Online Data: Understanding Preference Fine-tuning via Coverage cites Training language models to follow instructions with human feedback
- Prompt Curriculum Learning for Efficient LLM Post-Training cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Prompt Curriculum Learning for Efficient LLM Post-Training cites Training language models to follow instructions with human feedback
- Prompt Curriculum Learning for Efficient LLM Post-Training cites Accelerating RL for LLM Reasoning with Optimal Advantage Regression
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Cited by top-tier papers40
- What Makes a Reward Model a Good Teacher? An Optimization PerspectiveNoam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei et al.NeurIPS 2025 · 73 citations
- DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOJinyoung Park, Jeehye Na, Jinyoung Kim, Hyunwoo J. KimNeurIPS 2025 · 64 citations
- The Importance of Online Data: Understanding Preference Fine-tuning via CoverageYuda Song, Gokul Swamy, Aarti Singh, J. Andrew Bagnell et al.NeurIPS 2024 · 63 citations
- Prompt Curriculum Learning for Efficient LLM Post-TrainingZhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims et al.ICLR 2026 · 44 citations
- Accelerating RL for LLM Reasoning with Optimal Advantage RegressionKianté Brantley, Mingyu Chen, Zhaolin Gao, Jason D. Lee et al.NeurIPS 2025 · 31 citations
Builds on35
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
Related papers
- Rethinking the Trust Region in LLM Reinforcement LearningPenghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang et al.ICML 2026 · 22 citations
- Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMsArash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee et al.ACL 2024 · 20 citations
- Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement LearningSai Ashish Somayajula, Bokai Hu, Qi Cao, Xin Pan et al.EMNLP Findings 2025 · 1 citation
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language ModelsZiniu Li, Tian Xu, Yushun Zhang, Zhihang Lin et al.ICML 2024 · 165 citations
- Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition FunctionsSimon Matrenok, Skander Moalla, Caglar GulcehreNeurIPS 2025 · 6 citations
