Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, Kaixiang Lin
Abstract
Large Language Models (LLMs) are increasingly used as chatbots, yet their ability to personalize responses to user preferences remains limited. We introduce PREFEVAL, a benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in a long-context conversational setting. PREFEVAL comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PREFEVAL contains user personalization or preference information in both explicit and implicit forms, and evaluates LLM performance using a generation and a classification task. With PREFEVAL, we evaluated the aforementioned preference following capabilities of 10 open-source and proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval-augmented generation methods. Our benchmarking effort reveals that state-of-the-art LLMs face significant challenges in proactively following users' preferences during conversations. In particular, in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (∼3k tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations. Furthermore, we show that fine-tuning on PREFEVAL significantly improves performance. We believe PREFEVAL serves as a valuable resource for measuring, understanding, and enhancing LLMs' preference following abilities, paving the way for personalized conversational agents. Our code and dataset are available at https://prefeval.github.io/.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
22 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning cites Training language models to follow instructions with human feedback
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment cites Training language models to follow instructions with human feedback
- PersonaVLM: Long-Term Personalized Multimodal LLMs cites Training language models to follow instructions with human feedback
- PrefDisco: Benchmarking Proactive Personalized Reasoning cites Training language models to follow instructions with human feedback
- Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generation cites Training language models to follow instructions with human feedback
Cited by top-tier papers22
- Consistently Simulating Human Personas with Multi-Turn Reinforcement LearningMarwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff et al.NeurIPS 2025 · 51 citations
- Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form GenerationChengbing Wang, Yang Zhang, Wenjie Wang, Xiaoyan Zhao et al.ICLR 2026 · 35 citations
- Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized AlignmentWeixiang Zhao, Xingyu Sui, Yulin Hu, Jiahe Guo et al.NeurIPS 2025 · 34 citations
- PersonaVLM: Long-Term Personalized Multimodal LLMsChang Nie, Chaoyou Fu, Yifan Zhang, Haihua Yang et al.CVPR 2026 · 11 citations
- PrefDisco: Benchmarking Proactive Personalized ReasoningShuyue Stella Li, Avinandan Bose, Faeze Brahman, Simon S. Du et al.ICLR 2026 · 8 citations
Builds on19
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng et al.ICLR 2024 · 858 citations
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation DatasetLianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li et al.ICLR 2024 · 419 citations
Related papers
- ChildEval: WHEN LARGE LANGUAGE MODELS MEET CHILDREN'S PERSONALITIESYanyan Luo, Xue Han, Chunxu Zhao, Ruiqiao Bai et al.ACL Findings 2026
- Towards Preference Following in Tool Calling Language AgentsZhi-Yuan Chen, Siyu Lu, Qianlong Xie, Xingxing Wang et al.ACL Findings 2026
- Benchmarking and Improving LLM Robustness for Personalized GenerationChimaobi Okite, Naihao Deng, Kiran Bodipati, Huaidian Hou et al.EMNLP Findings 2025
- RefuteBench: Evaluating Refuting Instruction-Following for Large Language ModelsJianhao Yan, Yun Luo, Yue ZhangACL Findings 2024 · 2 citations
- Towards Proactive Personalization through Profile Customization for Individual Users in DialoguesXiaotian Zhang, Yuan Wang, Ruizhe Chen, Zeya Wang et al.ACL Findings 2026
