Lune

EMNLP Findings2024

Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

Kyuyoung Kim, Ah Jeong Seo, Hao Liu, Jinwoo Shin, Kimin Lee

Abstract

Large language models (LLMs) fine-tuned with alignment techniques, such as reinforcement learning from human feedback, have been instrumental in developing some of the most capable AI systems to date. Despite their success, existing methods typically rely on simple binary labels, such as those indicating preferred outputs in pairwise preferences, which fail to capture the subtle differences in relative quality between pairs. To address this limitation, we introduce an approach called Margin Matching Preference Optimization (MMPO), which incorporates relative quality margins into optimization, leading to improved LLM policies and reward models. Specifically, given quality margins in pairwise preferences, we design soft target probabilities based on the Bradley-Terry model, which are then used to train models with the standard cross-entropy objective. Experiments with both human and AI feedback data demonstrate that MMPO consistently outperforms baseline methods, often by a substantial margin, on popular benchmarks including MT-bench and RewardBench. Notably, the 7B model trained with MMPO achieves state-of-the-art performance on RewardBench as of June 2024, outperforming other models of the same scale. Our analysis also shows that MMPO is more robust to overfitting, leading to better-calibrated models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ComPO: Preference Alignment via Comparison Oracles cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • ComPO: Preference Alignment via Comparison Oracles cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • ComPO: Preference Alignment via Comparison Oracles cites Language Models are Few-Shot Learners
  • Margin-Aware Preference Optimization for Aligning Diffusion Models Without Reference cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Margin-Aware Preference Optimization for Aligning Diffusion Models Without Reference cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
  • Noisy Pairwise-Comparison Random Search for Smooth Nonconvex Optimization cites ComPO: Preference Alignment via Comparison Oracles
  • Noisy Pairwise-Comparison Random Search for Smooth Nonconvex Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Refining Language Model Anonymizers via Adversarial Distillation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Refining Language Model Anonymizers via Adversarial Distillation cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Margin-Aware Preference Optimization for Aligning Diffusion Models Without ReferenceJiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul et al.AAAI 2026 · 43 citations
  2. ComPO: Preference Alignment via Comparison OraclesPeter Chen, Xi Chen, Wotao Yin, Tianyi LinNeurIPS 2025 · 20 citations
  3. Pairwise Calibrated Rewards for Pluralistic AlignmentDaniel Halpern, Evi Micha, Ariel D. Procaccia, Itai ShapiraNeurIPS 2025 · 15 citations
  4. Self-Refining Language Model Anonymizers via Adversarial DistillationKyuyoung Kim, Hyunjun Jeon, Jinwoo ShinNeurIPS 2025 · 7 citations
  5. Noisy Pairwise-Comparison Random Search for Smooth Nonconvex OptimizationTaha EL BAKKALI EL KADI, Rayane Bouftini, Richard Zhang, Omar SaadiICML 2026 · 1 citation

Builds on15

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations

Related papers

  1. RMO: Towards Better LLM Alignment via Reshaping Reward Margin DistributionsYanchi Ru, Yue Huang, Xiangliang ZhangAAAI 2026 · 1 citation
  2. Robust Preference Optimization via Dynamic Target MarginsJie Sun, Junkang Wu, Jiancan Wu, Zhibo Zhu et al.ACL Findings 2025 · 1 citation
  3. Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
  4. Token-level Direct Preference OptimizationYongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang et al.ICML 2024 · 136 citations
  5. APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal TransportZhuo Li, Yuege Feng, Dandan Guo, Jinpeng Hu et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines