ACL Findings2025
MWPO: Enhancing LLMs Performance through Multi-Weight Preference Strength and Length Optimization
Abstract
Direct Preference Optimization (DPO) have proposed offline alternatives to ReinforcementLearning from Human Feedback (RLHF).In DPO, each preference pair, which serves as the foundation for learning, is typically constructed by first generating multiple responses to the same instruction and then annotating them to indicate the preferred choice.However, when the responses are highly similar, the weak preference signal can introduce annotation noise, which may hinder model optimization.Additionally, DPO suffers from the drawback of over-optimizing for verbose generation.A potential reason is the presence of length bias in preference datasets, which can lead to length exploitation.To address these issues, we propose a DPO-based multi-weight preference strength and length optimization (MWPO) method.Specifically, we propose to reweight preference pairs based on implicit reward margins and response length margins, unifying them through a geometric mixture to generate synthetic weights for optimization.This method allows preference pairs with stronger preference signals or more favorable length feature to have a more pronounced impact on model parameters.Moreover, our method does not require additional annotators.We validate our method on models of four different scales across multiple benchmarks.Our method surpasses state-of-the-art (SOTA) baselines, outperforming DPO by up to 8.7% on AlpacaEval 2 while reducing generation length by 9.4% in the Mistral setting 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Measuring Massive Multitask Language Understanding
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Training language models to follow instructions with human feedback
- Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites Measuring Massive Multitask Language Understanding
- Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites SimPO: Simple Preference Optimization with a Reference-Free Reward
- Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs cites Training language models to follow instructions with human feedback
Cited by top-tier papers2
- Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMsShangpin Peng, Weinong Wang, Zhuotao Tian, Senqiao Yang et al.ICLR 2026 · 10 citations
- Multi-Frequency Contrastive Decoding: Alleviating Hallucinations for Large Vision-Language ModelsBingqian Liu, Fu Zhang, Guoqing Chen, Jingwei ChengEMNLP 2025
Builds on17
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
Related papers
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay PerspectiveRuichen Shao, Bei Li, Gangao Liu, Yang Chen et al.ICLR 2025
- Less is More: Improving LLM Alignment via Preference Data SelectionXun Deng, Han Zhong, Rui Ai, Fuli Feng et al.NeurIPS 2025 · 4 citations
- Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference OptimizationJian Li, Shenglin Yin, Yujia Zhang, Alan Zhao et al.EMNLP 2025
- WPO: Enhancing RLHF with Weighted Preference OptimizationWenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi et al.EMNLP 2024 · 2 citations
- Towards Bridging the Reward-Generation Gap in Direct Alignment AlgorithmsZeguan Xiao, Yun Chen, Jian Yang, Guanhua Chen et al.ACL Findings 2026
