Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning
Jiahui Li, Hanlin Zhang, Fengda Zhang, Tai-Wei Chang, Kun Kuang, Long Chen, Jun Zhou
摘要
Reinforcement learning from human feedback (RLHF) and AI-generated feedback (RLAIF) have become prominent techniques that significantly enhance the functionality of pre-trained language models (LMs). These methods harness feedback, sourced either from humans or AI, as direct rewards or to shape reward models that steer LM optimization. Nonetheless, the effective integration of rewards from diverse sources presents a significant challenge due to their disparate characteristics. To address this, recent research has developed algorithms incorporating strategies such as weighting, ranking, and constraining to handle this complexity. Despite these innovations, a bias toward disproportionately high rewards can still skew the reinforcement learning process and negatively impact LM performance. This paper explores a methodology for reward composition that enables simultaneous improvements in LMs across multiple dimensions. Inspired by fairness theory, we introduce a training algorithm that aims to reduce Disparity and enhance Stability among various rewards. Our method treats the aggregate reward as a dynamic weighted sum of individual rewards, with alternating updates to the weights and model parameters. For efficient and straightforward implementation, we employ an estimation technique rooted in the mirror descent method for weight updates, eliminating the need for gradient computations. The empirical results under various types of rewards across a wide range of scenarios demonstrate the effectiveness of our method. Los Angeles Lakers have won the NBA Finals a total of 17 times. In total, the Lakers have played the Boston Celtics a record 12 times in the finals. In 2017, the Lakers defeated the Orlando Magic in five games in the finals. How many times have the lakers won the finals? Question: Los Angeles Lakers have won the NBA Finals a total of 17 times.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 5 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models 引用了 Training language models to follow instructions with human feedback
- Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation 引用了 Training language models to follow instructions with human feedback
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model 引用了 Language Models are Few-Shot Learners
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model 引用了 Training language models to follow instructions with human feedback
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Towards Reward Fairness in RLHF: From a Resource Allocation Perspective 引用了 Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback 引用了 Language Models are Few-Shot Learners
- Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare Estimation 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare Estimation 引用了 Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
引用它的顶会 Paper5
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language ModelsChengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue 等ACL 2025 · 13 次引用
- Towards Reward Fairness in RLHF: From a Resource Allocation PerspectiveSheng Ouyang, Yulan Hu, Ge Chen, Qingyang Li 等ACL 2025 · 10 次引用
- RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward RedistributionJiahui Li, Lin Li, Tai-Wei Chang, Kun Kuang 等EMNLP 2025
- Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective GenerationNing Wang, Zhanyang Liu, Taotao Zhou, Xinrui Zhang 等ACL 2026
- Graph-Preference Learning: Debiasing Network-Sampled Human Feedback for Target Welfare EstimationGuangrui Fan, DanDan Liu, AZNUL SABRI, Pan LihuICML 2026
它引用的 Paper13
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida 等NeurIPS 2022 · 24,707 次引用
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning 等NeurIPS 2023 · 10,924 次引用
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger 等ICLR 2020 · 8,443 次引用
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 次引用
相关 Paper
- MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference AlignmentTianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin 等ICML 2025
- Nash Learning from Human FeedbackRémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar 等ICML 2024 · 212 次引用
- Reinforcement Learning with Supervised AlignmentJoão Luís Lins, Jia XuEMNLP Findings 2025 · 1 次引用
- Reward Learning from Multiple Feedback TypesYannick Metz, András Geiszl, Raphaël Baur, Mennatallah El-AssadyICLR 2025
- Confronting Reward Model Overoptimization with Constrained RLHFTed Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm 等ICLR 2024 · 89 次引用
