Lune

ACL Findings2024

LIRE: listwise reward enhancement for preference alignment

Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao

Abstract

Recently, tremendous strides have been made to align the generation of Large Language Models (LLMs) with human values to mitigate toxic or unhelpful content. Leveraging Reinforcement Learning from Human Feedback (RLHF) proves effective and is widely adopted by researchers. However, implementing RLHF is complex, and its sensitivity to hyperparameters renders achieving stable performance and scalability challenging. Furthermore, prevailing approaches to preference alignment primarily concentrate on pairwise comparisons, with limited exploration into multi-response scenarios, thereby overlooking the potential richness within the candidate pool. For the above reasons, we propose a new approach: Listwise Reward Enhancement for Preference Alignment (LIRE), a gradient-based reward optimization approach that incorporates the offline rewards of multiple responses into a streamlined listwise framework, thus eliminating the need for online sampling during training. LIRE is straightforward to implement, requiring minimal parameter tuning, and seamlessly aligns with the pairwise paradigm while naturally extending to multi-response scenarios. Moreover, we introduce a self-enhancement algorithm aimed at iteratively refining the reward during training. Our experiments demonstrate that LIRE consistently outperforms existing methods across several benchmarks on dialogue and summarization tasks, with good transferability to out-of-distribution data, assessed using proxy reward models and human annotators.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization cites Language Models are Few-Shot Learners
  • FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization cites Training language models to follow instructions with human feedback
  • On-the-fly Preference Alignment via Principle-Guided Decoding cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • On-the-fly Preference Alignment via Principle-Guided Decoding cites Training language models to follow instructions with human feedback
  • Permutative Preference Alignment from Listwise Ranking of Human Judgments cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models cites FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
  • Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models cites On-the-fly Preference Alignment via Principle-Guided Decoding
  • Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models cites Training language models to follow instructions with human feedback
  • Leveraging robust optimization for llm alignment under distribution shifts cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Leveraging robust optimization for llm alignment under distribution shifts cites Training language models to follow instructions with human feedback
  • RRHF: Rank Responses to Align Language Models with Human Feedback cites Language Models are Few-Shot Learners
  • RRHF: Rank Responses to Align Language Models with Human Feedback cites Training language models to follow instructions with human feedback
  • LIMA: Less Is More for Alignment cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Permutative Preference Alignment from Listwise Ranking of Human JudgmentsYang Zhao, Yixin Wang, Mingzhang YinEMNLP 2025 · 5 citations
  2. FlipGuard: Defending Preference Alignment against Update Regression with Constrained OptimizationMingye Zhu, Yi Liu, Quan Wang, Junbo Guo et al.EMNLP 2024 · 2 citations
  3. Leveraging robust optimization for llm alignment under distribution shiftsMingye Zhu, Yi Liu, Zheren Fu, Yongdong Zhang et al.NeurIPS 2025 · 2 citations
  4. Leveraging Importance Sampling to Detach Alignment Modules from Large Language ModelsYi Liu, Dianqing Liu, Mingye Zhu, Junbo Guo et al.NeurIPS 2025 · 1 citation
  5. On-the-fly Preference Alignment via Principle-Guided DecodingMingye Zhu, Yi Liu, Lei Zhang, Junbo Guo et al.ICLR 2025

Builds on12

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations
  5. RRHF: Rank Responses to Align Language Models with Human FeedbackHongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang et al.NeurIPS 2023 · 515 citations

Related papers

  1. Nash Learning from Human FeedbackRémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar et al.ICML 2024 · 212 citations
  2. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI FeedbackHarrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard et al.ICML 2024 · 598 citations
  3. ALaRM: Align Language Models via Hierarchical Rewards ModelingYuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang et al.ACL Findings 2024 · 5 citations
  4. Preference Ranking Optimization for Human AlignmentFeifan Song, Bowen Yu, Minghao Li, Haiyang Yu et al.AAAI 2024 · 357 citations
  5. GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNetsOh Joon Kwon, Daiki E. Matsunaga, Kee-Eung KimEMNLP 2024 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines