Lune

EMNLP2024Top-tier venue

Who is better at math, Jenny or Jingzhen? Uncovering Stereotypes in Large Language Models

Zara Siddique, Liam D. Turner, Luis Espinosa Anke

Abstract

Large language models (LLMs) have been shown to propagate and amplify harmful stereotypes, particularly those that disproportionately affect marginalised communities. To understand the effect of these stereotypes more comprehensively, we introduce GlobalBias, a dataset of 876k sentences incorporating 40 distinct gender-by-ethnicity groups alongside descriptors typically used in bias literature, which enables us to study a broad set of stereotypes from around the world. We use GlobalBias to directly probe a suite of LMs via perplexity, which we use as a proxy to determine how certain stereotypes are represented in the model’s internal representations. Following this, we generate character profiles based on given names and evaluate the prevalence of stereotypes in model outputs. We find that the demographic groups associated with various stereotypes remain consistent across model likelihoods and model outputs. Furthermore, larger models consistently display higher levels of stereotypical outputs, even when explicitly instructed not to.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language Models cites BBQ: A hand-built bias benchmark for question answering
  • Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models cites BBQ: A hand-built bias benchmark for question answering
  • Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models cites Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. KLAAD: Refining Attention Mechanisms to Reduce Societal Bias in Generative Language ModelsSeorin Kim, Dongyoung Lee, Jaejin LeeEMNLP 2025

Builds on11

  1. Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language ModelsHannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal et al.NeurIPS 2021 · 243 citations
  2. BBQ: A hand-built bias benchmark for question answeringAlicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar et al.ACL Findings 2022 · 186 citations
  3. Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, Katrin KirchhoffACL 2020 · 167 citations
  4. UL2: Unifying Language Learning ParadigmsYi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia et al.ICLR 2023 · 97 citations
  5. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language ModelsMyra Cheng, Esin Durmus, Dan JurafskyACL 2023 · 89 citations

Related papers

  1. StereoSet: Measuring stereotypical bias in pretrained language modelsMoin Nadeem, Anna Bethke, Siva ReddyACL 2021
  2. Intersectional Stereotypes in Large Language Models: Dataset and AnalysisWeicheng Ma, Brian Chiang, Tong Wu, Lili Wang et al.EMNLP Findings 2023 · 7 citations
  3. LLMs Reproduce Stereotypes of Sexual and Gender MinoritiesRuby Ostrow, Adam LopezEMNLP Findings 2025 · 7 citations
  4. "I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor DatasetEric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani et al.EMNLP 2022 · 56 citations
  5. Deus Ex Machina and Personas from Large Language Models: Investigating the Composition of AI-Generated Persona DescriptionsJoni Salminen, Chang Liu, Wenjing Pian, Jianxing Chi et al.CHI 2024 · 55 citations
Dusk over the sea between two cliffs drawn in fine vertical lines