Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen
Abstract
Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans express multiple, often conflicting objectives, such as helpfulness and harmlessness, with no natural scalarization. We study the multi-objective preference alignment problem, where a policy must balance several objectives simultaneously. We propose Multi-Objective Preference Optimization (MOPO), a constrained KL-regularized framework that maximizes a primary objective while enforcing lower bounds on secondary objectives via tunable safety thresholds. MOPO operates directly on pairwise preferences without point-wise rewards, and admits simple closed-form iterative updates. Empirically, MOPO recovers Pareto-optimal policies on synthetic benchmarks and, when fine-tuned on human-preference data, yields multi-billion parameter models that achieve higher rewards and Pareto-dominate baselines, with stable and robust optimization dynamics.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Training language models to follow instructions with human feedback
- From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- MOA: Multi-Objective Alignment for Role-Playing Agents cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models cites LoRA: Low-Rank Adaptation of Large Language Models
- MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models cites Training language models to follow instructions with human feedback
- PICACO: Pluralistic In-Context Value Alignment via Total Correlation Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Cited by top-tier papers7
- From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image GenerationZiwei Huang, Yin Shu, Hao Fang, Quanyu Long et al.ACL 2026 · 4 citations
- MOA: Multi-Objective Alignment for Role-Playing AgentsChonghua Liao, Ke Wang, Yuchuan Wu, Ruoran Li et al.ACL 2026 · 4 citations
- MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative ModelsChieh-Yun Chen, Zhonghao Wang, Qi Chen, Zhifan Ye et al.CVPR 2026 · 3 citations
- Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM ReviewsHyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim et al.EMNLP 2025 · 2 citations
- PICACO: Pluralistic In-Context Value Alignment via Total Correlation OptimizationHan Jiang, Dongyao Zhu, Xiaoyuan Yi, Ziang Xiao et al.ICML 2026 · 2 citations
Builds on23
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human FeedbackYann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang et al.NeurIPS 2023 · 948 citations
Related papers
- Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference OptimizationZhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue et al.ACL Findings 2024 · 21 citations
- Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective AlignmentMoxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi et al.ACL Findings 2025 · 3 citations
- Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language ModelsChengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue et al.ACL 2025 · 13 citations
- Multiplayer Nash Preference OptimizationFang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang et al.ICLR 2026 · 8 citations
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
