Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning
Jiahui Li, Hanlin Zhang, Fengda Zhang, Tai-Wei Chang, Kun Kuang, Long Chen, Jun Zhou
Abstract
Reinforcement learning from human feedback (RLHF) and AI-generated feedback (RLAIF) have become prominent techniques that significantly enhance the functionality of pre-trained language models (LMs). These methods harness feedback, sourced either from humans or AI, as direct rewards or to shape reward models that steer LM optimization. Nonetheless, the effective integration of rewards from diverse sources presents a significant challenge due to their disparate characteristics. To address this, recent research has developed algorithms incorporating strategies such as weighting, ranking, and constraining to handle this complexity. Despite these innovations, a bias toward disproportionately high rewards can still skew the reinforcement learning process and negatively impact LM performance. This paper explores a methodology for reward composition that enables simultaneous improvements in LMs across multiple dimensions. Inspired by fairness theory, we introduce a training algorithm that aims to reduce Disparity and enhance Stability among various rewards. Our method treats the aggregate reward as a dynamic weighted sum of individual rewards, with alternating updates to the weights and model parameters. For efficient and straightforward implementation, we employ an estimation technique rooted in the mirror descent method for weight updates, eliminating the need for gradient computations. The empirical results under various types of rewards across a wide range of scenarios demonstrate the effectiveness of our method. Los Angeles Lakers have won the NBA Finals a total of 17 times. In total, the Lakers have played the Boston Celtics a record 12 times in the finals. In 2017, the Lakers defeated the Orlando Magic in five games in the finals. How many times have the lakers won the finals? Question: Los Angeles Lakers have won the NBA Finals a total of 17 times.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models cites Training language models to follow instructions with human feedback
- Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation cites Training language models to follow instructions with human feedback
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
- Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare Estimation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare Estimation cites Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
Cited by top-tier papers5
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language ModelsChengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue et al.ACL 2025 · 13 citations
- Towards Reward Fairness in RLHF: From a Resource Allocation PerspectiveSheng Ouyang, Yulan Hu, Ge Chen, Qingyang Li et al.ACL 2025 · 10 citations
- RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward RedistributionJiahui Li, Lin Li, Tai-Wei Chang, Kun Kuang et al.EMNLP 2025
- Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective GenerationNing Wang, Zhanyang Liu, Taotao Zhou, Xinrui Zhang et al.ACL 2026
- Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare EstimationGuangrui Fan, DanDan Liu, AZNUL SABRI, Pan LihuICML 2026
Builds on13
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 citations
Related papers
- MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference AlignmentTianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin et al.ICML 2025
- Nash Learning from Human FeedbackRémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar et al.ICML 2024 · 212 citations
- Reinforcement Learning with Supervised AlignmentJoão Luís Lins, Jia XuEMNLP Findings 2025 · 1 citation
- Reward Learning from Multiple Feedback TypesYannick Metz, András Geiszl, Raphaël Baur, Mennatallah El-AssadyICLR 2025
- Confronting Reward Model Overoptimization with Constrained RLHFTed Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm et al.ICLR 2024 · 89 citations
