ACL Findings2025
Stereotype or Personalization? User Identity Biases Chatbot Recommendations
Anjali Kantharuban, Jeremiah Milbauer, Maarten Sap, Emma Strubell, Graham Neubig
Abstract
While personalized recommendations are often desired by users, it can be difficult in practice to distinguish cases of bias from cases of personalization: we find that models generate racially stereotypical recommendations regardless of whether the user revealed their identity intentionally through explicit indications or unintentionally through implicit cues. We demonstrate that when people use large language models (LLMs) to generate recommendations, the LLMs produce responses that reflect both what the user wants and who the user is. We argue that chatbots ought to transparently indicate when recommendations are influenced by a user's revealed identity characteristics, but observe that they currently fail to do so. Our experiments show that even though a user's revealed identity significantly influences model recommendations (p < 0.001), model responses obfuscate this fact in response to user queries. This bias and lack of transparency occurs consistently across multiple popular consumer LLMs and for four American racial groups. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks cites Understanding the Impacts of Language Technologies' Performance Disparities on African American Language Speakers
- Understanding the Impacts of Language Technologies' Performance Disparities on African American Language Speakers cites Quantifying the Dialect Gap and its Correlates Across Languages
- Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs cites "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters
Cited by top-tier papers8
- Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMsAngelina Wang, Michelle Phan, Daniel E. Ho, Sanmi KoyejoACL 2025 · 17 citations
- Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning TasksFangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann et al.ACL 2025 · 14 citations
- One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM PersonalizationFranziska Weeber, Vera Neplenbroek, Jan Batzner, Sebastian PadóACL 2026 · 4 citations
- To Mask or to Mirror: Human-AI Alignment in Collective ReasoningCrystal Qian, Aaron T. Parisi, Clémentine Bouleau, Vivian Tsai et al.EMNLP 2025 · 1 citation
- Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real UsersNishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman et al.ACL 2026 · 1 citation
Builds on11
- (Un)informed Consent: Studying GDPR Consent Notices in the FieldChristine Utz, Martin Degeling, Sascha Fahl, Florian Schaub et al.CCS 2019 · 429 citations
- "Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference LettersYixin Wan, George Pu, Jiao Sun, Aparna Garimella et al.EMNLP Findings 2023 · 139 citations
- Do All Languages Cost the Same? Tokenization in the Era of Commercial Language ModelsOrevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai et al.EMNLP 2023 · 24 citations
- Understanding the Impacts of Language Technologies' Performance Disparities on African American Language SpeakersJay Cunningham, Su Lin Blodgett, Michael Madaio, Hal Daumé III et al.ACL Findings 2024 · 16 citations
- Quantifying the Dialect Gap and its Correlates Across LanguagesAnjali Kantharuban, Ivan Vulic, Anna KorhonenEMNLP Findings 2023 · 12 citations
Related papers
- Reading Between the Prompts: How Stereotypes Shape LLM's Implicit PersonalizationVera Neplenbroek, Arianna Bisazza, Raquel FernándezEMNLP 2025
- Presumed Cultural Identity: How Names Shape LLM ResponsesSiddhesh Pawar, Arnav Arora, Lucie-Aimée Kaffee, Isabelle AugensteinEMNLP Findings 2025 · 2 citations
- Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMsShashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan et al.ICLR 2024 · 212 citations
- Deus Ex Machina and Personas from Large Language Models: Investigating the Composition of AI-Generated Persona DescriptionsJoni Salminen, Chang Liu, Wenjing Pian, Jianxing Chi et al.CHI 2024 · 55 citations
- Transparent and Scrutable Recommendations Using Natural Language User ProfilesJerome Ramos, Hossein A. Rahmani, Xi Wang, Xiao Fu et al.ACL 2024
