Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Myra Cheng, Esin Durmus, Dan Jurafsky
Abstract
To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs. Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexicon or data labeling. Grounded in the sociolinguistic concept of markedness (which characterizes explicitly linguistically marked categories versus unmarked defaults), our proposed method is twofold: 1) prompting an LLM to generate personas, i.e., natural language descriptions, of the target demographic group alongside personas of unmarked, default groups; 2) identifying the words that significantly distinguish personas of the target group from corresponding unmarked ones. We find that the portrayals generated by GPT-3.5 and GPT-4 contain higher rates of racial stereotypes than human-written portrayals using the same prompts. The words distinguishing personas of marked (non-white, non-male) groups reflect patterns of othering and exoticizing these demographics. An intersectional lens further reveals tropes that dominate portrayals of marginalized groups, such as tropicalism and the hypersexualization of minoritized women. These representational harms have concerning implications for downstream applications like story generation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
51 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers51
- SOTOPIA: Interactive Evaluation for Social Intelligence in Language AgentsXuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang et al.ICLR 2024 · 288 citations
- Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMsShashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan et al.ICLR 2024 · 212 citations
- Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with HumansJen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren et al.NeurIPS 2024 · 63 citations
- Rehearsal: Simulating Conflict to Teach Conflict ResolutionOmar Shaikh, Valentino Emil Chai, Michele Gelfand, Diyi Yang et al.CHI 2024 · 62 citations
- Deus Ex Machina and Personas from Large Language Models: Investigating the Composition of AI-Generated Persona DescriptionsJoni Salminen, Chang Liu, Wenjing Pian, Jianxing Chi et al.CHI 2024 · 55 citations
Builds on12
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model CapabilitiesMina Lee, Percy Liang, Qian YangCHI 2022 · 340 citations
- Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language ModelsHannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal et al.NeurIPS 2021 · 243 citations
- BBQ: A hand-built bias benchmark for question answeringAlicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar et al.ACL Findings 2022 · 186 citations
- Long Time No See! Open-Domain Conversation with Long-Term Persona MemoryXinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu et al.ACL Findings 2022 · 65 citations
Related papers
- Simulating Identity, Propagating Bias: Abstraction and Stereotypes in LLM-Generated TextPia Sommerauer, Giulia Rambelli, Tommaso CaselliEMNLP Findings 2025 · 3 citations
- The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language ModelsMarlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers et al.EMNLP Findings 2025 · 11 citations
- Intersectional Stereotypes in Large Language Models: Dataset and AnalysisWeicheng Ma, Brian Chiang, Tong Wu, Lili Wang et al.EMNLP Findings 2023 · 7 citations
- Reading Between the Prompts: How Stereotypes Shape LLM's Implicit PersonalizationVera Neplenbroek, Arianna Bisazza, Raquel FernándezEMNLP 2025
- Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue SystemsYixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng et al.EMNLP Findings 2023 · 8 citations
