Lune

EMNLP Findings2024

Reward Difference Optimization For Sample Reweighting In Offline RLHF

Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, Cam-Tu Nguyen

Abstract

With the rapid advances in Large Language Models (LLMs), aligning LLMs with human preferences become increasingly important. Although Reinforcement Learning with Human Feedback (RLHF) proves effective, it is complicated and highly resource-intensive. As such, offline RLHF has been introduced as an alternative solution, which directly optimizes LLMs with ranking losses on a fixed preference dataset. Current offline RLHF only captures the "ordinal relationship" between responses, overlooking the crucial aspect of "how much" one is preferred over the others. To address this issue, we propose a simple yet effective solution called Reward Difference Optimization, shorted as RDO. Specifically, we introduce reward difference coefficients to reweigh sample pairs in offline RLHF. We then develop a difference model which captures rich interactions between a pair of responses for predicting these difference coefficients. Experiments with 7B LLMs on the HH and TL;DR datasets substantiate the effectiveness of our method in both automatic metrics and human evaluation, thereby highlighting its potential for aligning LLMs with human intent and values.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Model Alignment as Prospect Theoretic Optimization cites Training language models to follow instructions with human feedback
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning cites MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy
  • MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy cites ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning
  • RePO: Understanding Preference Learning Through ReLU-Based Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • RePO: Understanding Preference Learning Through ReLU-Based Optimization cites Training language models to follow instructions with human feedback
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites RRHF: Rank Responses to Align Language Models with Human Feedback
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Training language models to follow instructions with human feedback
  • Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites SimPO: Simple Preference Optimization with a Reference-Free Reward
  • Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites Training language models to follow instructions with human feedback
  • RRHF: Rank Responses to Align Language Models with Human Feedback cites Training language models to follow instructions with human feedback
  • Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? cites Model Alignment as Prospect Theoretic Optimization
  • Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced PolicyShaoxiong Zhan, Yanlin Lai, Ziyu Lu, Dahua Lin et al.AAAI 2026 · 17 citations
  2. Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMsShangpin Peng, Weinong Wang, Zhuotao Tian, Senqiao Yang et al.ICLR 2026 · 10 citations
  3. RePO: Understanding Preference Learning Through ReLU-Based OptimizationJunkang Wu, Kexin Huang, Xue Wang, Jinyang Gao et al.NeurIPS 2025 · 4 citations
  4. ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuningYiming Du, Yifan Xiang, Bin Liang, Dahua Lin et al.EMNLP 2025 · 1 citation
  5. Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang et al.ICML 2025

Builds on16

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
  4. Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
  5. RRHF: Rank Responses to Align Language Models with Human FeedbackHongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang et al.NeurIPS 2023 · 515 citations

Related papers

  1. Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree AwarenessJian Li, Haojing Huang, Yujia Zhang, Pengfei Xu et al.EMNLP Findings 2024 · 2 citations
  2. Direct Preference Optimization with an OffsetAfra Amini, Tim Vieira, Ryan CotterellACL Findings 2024 · 13 citations
  3. Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
  4. Preference Ranking Optimization for Human AlignmentFeifan Song, Bowen Yu, Minghao Li, Haiyang Yu et al.AAAI 2024 · 357 citations
  5. LIRE: listwise reward enhancement for preference alignmentMingye Zhu, Yi Liu, Lei Zhang, Junbo Guo et al.ACL Findings 2024 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines