Lune

ICLR2025Top-tier venue

Strong Preferences Affect the Robustness of Preference Models and Value Alignment

Ziwei Xu, Mohan S. Kankanhalli

Abstract

Value alignment, which aims to ensure that large language models (LLMs) and other AI agents behave in accordance with human values, is critical for ensuring safety and trustworthiness of these systems. A key component of value alignment is the modeling of human preferences as a representation of human values. In this paper, we investigate the robustness of value alignment by examining the sensitivity of preference models. Specifically, we ask: how do changes in the probabilities of some preferences affect the predictions of these models for other preferences? To answer this question, we theoretically analyze the robustness of widely used preference models by examining their sensitivities to minor changes in preferences they model. Our findings reveal that, in the Bradley-Terry and the Placket-Luce model, the probability of a preference can change significantly as other preferences change, especially when these preferences are dominant (i.e., with probabilities near 0 or 1). We identify specific conditions where this sensitivity becomes significant for these models and discuss the practical implications for the robustness and safety of value alignment in AI systems.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation cites Training language models to follow instructions with human feedback
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning cites Training language models to follow instructions with human feedback
  • Mitigating Reward Hacking in LLM-based Recommendation: A Preference Optimization Approach cites Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
  • Mitigating Reward Hacking in LLM-based Recommendation: A Preference Optimization Approach cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Mitigating Reward Hacking in LLM-based Recommendation: A Preference Optimization Approach cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. ResponseRank: Data-Efficient Reward Modeling through Preference Strength LearningTimo Kaufmann, Yannick Metz, Daniel A. Keim, Eyke HüllermeierNeurIPS 2025 · 2 citations
  2. Mitigating Reward Hacking in LLM-based Recommendation: A Preference Optimization ApproachHeyu Chen, Junkang Wu, Guoqing Hu, Kexin Huang et al.ICML 2026

Builds on11

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos et al.ICML 2024 · 1,212 citations
  4. Defining and Characterizing Reward GamingJoar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David KruegerNeurIPS 2022 · 466 citations
  5. Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine TranslationHaoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan et al.ICML 2024 · 447 citations

Related papers

  1. Doubly Robust Alignment for Large Language ModelsErhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu et al.NeurIPS 2025 · 14 citations
  2. HPS: Hard Preference Sampling for Human Preference AlignmentXiandong Zou, Wanyu Lin, Yuchen Li, Pan ZhouICML 2025
  3. Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language ModelsRei Higuchi, Taiji SuzukiICML 2025
  4. Understanding the Learning Dynamics of Alignment with Human FeedbackShawn Im, Yixuan LiICML 2024 · 18 citations
  5. Leveraging robust optimization for llm alignment under distribution shiftsMingye Zhu, Yi Liu, Zheren Fu, Yongdong Zhang et al.NeurIPS 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines