Pairwise Calibrated Rewards for Pluralistic Alignment
Daniel Halpern, Evi Micha, Ariel D. Procaccia, Itai Shapira
Abstract
Current alignment pipelines presume a single, universal notion of desirable behavior. However, human preferences often diverge across users, contexts, and cultures. As a result, disagreement collapses into the majority signal and minority perspectives are discounted. To address this, we propose reflecting diverse human preferences through a distribution over multiple reward functions, each inducing a distinct aligned policy. The distribution is learned directly from pairwise preference without annotator identifiers or predefined groups. Instead, annotator disagreements are treated as informative soft labels. Our central criterion is pairwise calibration: for every pair of candidate responses, the proportion of reward functions preferring one response matches the fraction of annotators with that preference. We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions. Empirically, we introduce and validate a practical training heuristic to learn such ensembles, and demonstrate its effectiveness through improved calibration, implying a more faithful representation of pluralistic values.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework cites Training language models to follow instructions with human feedback
- Understanding the Effects of RLHF on LLM Generalisation and Diversity cites Training language models to follow instructions with human feedback
- Calibrated Preference Learning: The Case of Label Ranking cites Training language models to follow instructions with human feedback
- Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards cites Training language models to follow instructions with human feedback
- Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards cites Fine-tuning language models to find agreement among humans with diverse preferences
- Whose Opinions Do Language Models Reflect? cites Training language models to follow instructions with human feedback
- When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models cites Training language models to follow instructions with human feedback
- How RLHF Amplifies Sycophancy cites Training language models to follow instructions with human feedback
- Robust AI Evaluation through Maximal Lotteries cites Training language models to follow instructions with human feedback
- Fine-tuning language models to find agreement among humans with diverse preferences cites Training language models to follow instructions with human feedback
Cited by top-tier papers6
- How RLHF Amplifies SycophancyItai Shapira, Gerdus Benade, Ariel ProcacciaICML 2026 · 16 citations
- Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic FrameworkKihyun Kim, Jiawei Zhang, Asuman Ozdaglar, Pablo A. ParriloICLR 2026 · 5 citations
- Robust AI Evaluation through Maximal LotteriesHadi Khalaf, Serena Wang, Daniel Halpern, Itai Shapira et al.ICML 2026 · 2 citations
- Calibrated Preference Learning: The Case of Label RankingSanto Thies, Viktor Bengs, Timo Kaufmann, Sebastian Vollmer et al.ICML 2026
- When Distance Distracts: Representation Distance Bias in BT-Loss for Reward ModelsTong Xie, Ching-Yuan Bai, Yuanhao Ban, Yunqi Hong et al.ICML 2026
Builds on31
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Whose Opinions Do Language Models Reflect?Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee et al.ICML 2023 · 764 citations
- Fine-tuning language models to find agreement among humans with diverse preferencesMichiel A. Bakker, Martin J. Chadwick, Hannah Sheahan, Michael Henry Tessler et al.NeurIPS 2022 · 349 citations
- Understanding the Effects of RLHF on LLM Generalisation and DiversityRobert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina et al.ICLR 2024 · 332 citations
- Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardsAlexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya et al.NeurIPS 2023 · 295 citations
Related papers
- Direct Alignment with Heterogeneous PreferencesAli Shirali, Arash Nasr-Esfahany, Abdullah Omar Alomar, Parsa Mirtaheri et al.NeurIPS 2025 · 26 citations
- Geometric-Averaged Preference Optimization for Soft Preference LabelsHiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo et al.NeurIPS 2024 · 24 citations
- Diverse Preference Learning for Capabilities and AlignmentStewart Slocum, Asher Parker-Sartori, Dylan Hadfield-MenellICLR 2025
- Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language ModelsYanggan Gu, Junzhuo Li, Sirui Huang, Xin Zou et al.ACL Findings 2025
- No Preference Left Behind: Group Distributional Preference OptimizationBinwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang et al.ICLR 2025
