ACL Findings2026
Personalized Benchmarking: Evaluating LLMs by Individual Preferences
Abstract
With the rise in capabilities of large language models (LLMs) and their deployment in realworld tasks, evaluating LLM alignment with human preferences has become an important challenge. Current benchmarks average preferences across all users to compute aggregate ratings, overlooking individual user preferences when establishing model rankings. Since users have varying preferences in different contexts, we call for personalized LLM benchmarks that rank models according to individual needs. We compute personalized model rankings using ELO ratings and Bradley-Terry coefficients for 115 active Chatbot Arena users and analyze how user query characteristics (topics and writing style) relate to LLM ranking variations. We demonstrate that individual rankings of LLM models diverge dramatically from aggregate LLM rankings, with Bradley-Terry correlations averaging only ρ = 0.04 (57% of users show near-zero or negative correlation) and ELO ratings showing moderate correlation (ρ = 0.43). Through topic modeling and style analysis, we find users exhibit substantial heterogeneity in topical interests and communication styles, influencing their model preferences. We further show that a compact combination of topic and style features provides a useful feature space for predicting user-specific model rankings. Our results provide strong quantitative evidence that aggregate benchmarks fail to capture individual preferences for most users, and highlight the importance of developing personalized benchmarks that rank LLM models according to individual user preferences.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- Expanding Explainability: Towards Social Transparency in AI systemsUpol Ehsan, Q. Vera Liao, Michael J. Muller, Mark O. Riedl et al.CHI 2021 · 505 citations
- FASTopic: Pretrained Transformer is a Fast, Adaptive, Stable, and Transferable Topic ModelXiaobao Wu, Thong Nguyen, Delvin Zhang, William Yang Wang et al.NeurIPS 2024 · 67 citations
- Learning Interpretable Style Embeddings via Prompting LLMsAjay Patel, Delip Rao, Ansh Kothary, Kathleen R. McKeown et al.EMNLP Findings 2023 · 10 citations
- An Empirical Analysis of the Writing Styles of Persona-Assigned LLMsManuj Malik, Jing Jiang, Kian Ming A. ChaiEMNLP 2024 · 2 citations
- From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder PipelineTianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap et al.ICML 2025
Related papers
- Dropping Just a Handful of Preferences Can Change Top Large Language Model RankingsJenny Y. Huang, Yunyi Shen, Dennis Wei, Tamara BroderickICLR 2026 · 8 citations
- Prompt-to-Leaderboard: Prompt-Adaptive LLM EvaluationsEvan Frick, Connor Chen, Joseph Tennyson, Tianle Li et al.ICML 2025
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos et al.ICML 2024 · 1,212 citations
- Prediction-Powered Ranking of Large Language ModelsIvi Chatzi, Eleni Straitouri, Suhas Thejaswi, Manuel Gomez RodriguezNeurIPS 2024 · 34 citations
- PersonalLLM: Tailoring LLMs to Individual PreferencesThomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li et al.ICLR 2025
