Lune

ICML2026Top-tier venue

Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen

Abstract

Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans express multiple, often conflicting objectives, such as helpfulness and harmlessness, with no natural scalarization. We study the multi-objective preference alignment problem, where a policy must balance several objectives simultaneously. We propose Multi-Objective Preference Optimization (MOPO), a constrained KL-regularized framework that maximizes a primary objective while enforcing lower bounds on secondary objectives via tunable safety thresholds. MOPO operates directly on pairwise preferences without point-wise rewards, and admits simple closed-form iterative updates. Empirically, MOPO recovers Pareto-optimal policies on synthetic benchmarks and, when fine-tuned on human-preference data, yields multi-billion parameter models that achieve higher rewards and Pareto-dominate baselines, with stable and robust optimization dynamics.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback cites Training language models to follow instructions with human feedback
  • From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • MOA: Multi-Objective Alignment for Role-Playing Agents cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models cites Training language models to follow instructions with human feedback
  • PICACO: Pluralistic In-Context Value Alignment via Total Correlation Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image GenerationZiwei Huang, Yin Shu, Hao Fang, Quanyu Long et al.ACL 2026 · 4 citations
  2. MOA: Multi-Objective Alignment for Role-Playing AgentsChonghua Liao, Ke Wang, Yuchuan Wu, Ruoran Li et al.ACL 2026 · 4 citations
  3. MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative ModelsChieh-Yun Chen, Zhonghao Wang, Qi Chen, Zhifan Ye et al.CVPR 2026 · 3 citations
  4. Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM ReviewsHyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim et al.EMNLP 2025 · 2 citations
  5. PICACO: Pluralistic In-Context Value Alignment via Total Correlation OptimizationHan Jiang, Dongyao Zhu, Xiaoyuan Yi, Ziang Xiao et al.ICML 2026 · 2 citations

Builds on23

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
  5. AlpacaFarm: A Simulation Framework for Methods that Learn from Human FeedbackYann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang et al.NeurIPS 2023 · 948 citations

Related papers

  1. Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference OptimizationZhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue et al.ACL Findings 2024 · 21 citations
  2. Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective AlignmentMoxin Li, Yuantao Zhang, Wenjie Wang, Wentao Shi et al.ACL Findings 2025 · 3 citations
  3. Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language ModelsChengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue et al.ACL 2025 · 13 citations
  4. Multiplayer Nash Preference OptimizationFang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang et al.ICLR 2026 · 8 citations
  5. Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
Dusk over the sea between two cliffs drawn in fine vertical lines