EMNLP Findings2025
When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning
Yijiang River Dong, Tiancheng Hu, Yinhong Liu, Ahmet Üstün, Nigel Collier
Abstract
While Reinforcement Learning from Human Feedback (RLHF) is widely used to align Large Language Models (LLMs) with human preferences, it typically assumes homogeneous preferences across users, overlooking diverse human values and minority viewpoints. Although personalized preference learning addresses this by tailoring separate preferences for individual users, the field lacks standardized methods to assess its effectiveness. We present a multifaceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence. Through extensive experiments comparing eight personalization methods across three preference datasets, we demonstrate that performance differences between methods could reach 36% when users strongly disagree, and personalization can introduce up to 20% safety misalignment. These findings highlight the critical need for holistic evaluation approaches to advance the development of more effective and inclusive preference learning systems.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas cites Whose Opinions Do Language Models Reflect?
- Whose Opinions Do Language Models Reflect? cites Training language models to follow instructions with human feedback
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF cites Learning to summarize with human feedback
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF cites Training language models to follow instructions with human feedback
Cited by top-tier papers3
- Value of Information: A Framework for Human-Agent CommunicationYijiang River Dong, Tiancheng Hu, Zheng Hui, Caiqi Zhang et al.ACL 2026 · 8 citations
- Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect PersonasLuke Guerdan, Justin Whitehouse, Kimberly Truong, Ken Holstein et al.ICLR 2026 · 8 citations
- Personalization up to a Point: Why Personalized Content Moderation Needs Boundaries, and How We Can Enforce ThemEmanuele Moscato, Tiancheng Hu, Matthias Orlikowski, Paul Röttger et al.EMNLP 2025
Builds on18
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Whose Opinions Do Language Models Reflect?Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee et al.ICML 2023 · 764 citations
- Beyond Memorization: Violating Privacy via Inference with Large Language ModelsRobin Staab, Mark Vero, Mislav Balunovic, Martin T. VechevICLR 2024 · 211 citations
- Learning to summarize with human feedbackNisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler et al.NeurIPS 2020 · 124 citations
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFAnand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-MenellICLR 2024 · 112 citations
Related papers
- Unintended Impacts of LLM Alignment on Global RepresentationMichael J. Ryan, William Barr Held, Diyi YangACL 2024
- Doubly Robust Alignment for Large Language ModelsErhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu et al.NeurIPS 2025 · 14 citations
- Aligning Large Language Models with Human Preferences through Representation EngineeringWenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li et al.ACL 2024 · 4 citations
- Safe RLHF: Safe Reinforcement Learning from Human FeedbackJosef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji et al.ICLR 2024 · 656 citations
- On a Connection Between Imitation Learning and RLHFTeng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen et al.ICLR 2025
