EMNLP Findings2025
Neutral Is Not Unbiased: Evaluating Implicit and Intersectional Identity Bias in LLMs Through Structured Narrative Scenarios
Saba Ghanbari Haez, Mauro Dragoni
Abstract
Large Language Models often reproduce societal biases, yet most evaluations overlook how such biases evolve across nuanced contexts or intersecting identities. We introduce a scenario-based evaluation framework built on 100 narrative tasks, designed to be neutral at baseline and systematically modified with gender and age cues. Grounded in the theory of Normative-Narrative Scenarios, our approach provides ethically coherent and socially plausible settings for probing model behavior. Analyzing responses from five leading LLMs-GPT-4o, LLaMA 3.1, Qwen2.5, Phi-4, and Mistral-using Critical Discourse Analysis and quantitative linguistic metrics, we find consistent evidence of bias. Gender emerges as the dominant axis of bias, with intersectional cues (e.g., age and gender combined) further intensifying disparities. Our results underscore the value of dynamic narrative progression for detecting implicit, systemic biases in Large Language Models.
Ask Lune
Ask anything. The answer quotes the paper.
Lune reads all of it and the papers around it, then shows you the passages behind every answer.
Builds on11
- Towards Controllable Biases in Language GenerationEmily Sheng, Kai-Wei Chang, Prem Natarajan, Nanyun PengEMNLP Findings 2020 · 89 citations
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative ModelsMahammed Kamruzzaman, Md. Minul Islam Shovon, Gene Louis KimACL Findings 2024 · 18 citations
- Queens are Powerful too: Mitigating Gender Bias in Dialogue GenerationEmily Dinan, Angela Fan, Adina Williams, Jack Urbanek et al.EMNLP 2020 · 14 citations
- Intersectional Stereotypes in Large Language Models: Dataset and AnalysisWeicheng Ma, Brian Chiang, Tong Wu, Lili Wang et al.EMNLP Findings 2023 · 7 citations
- Gender Bias in Decision-Making with Large Language Models: A Study of Relationship ConflictsSharon Levy, William D. Adler, Tahilin Sanchez Karver, Mark Dredze et al.EMNLP Findings 2024 · 7 citations
Related papers
- Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted TextTunazzina IslamACL Findings 2026
- Evaluating Gender Bias of LLMs in Making Morality JudgementsDivij Bajaj, Yuanyuan Lei, Jonathan Tong, Ruihong HuangEMNLP Findings 2024 · 3 citations
- Large Language Models Still Exhibit Bias in Long TextWonje Jeung, Dongjae Jeon, Ashkan Yousefpour, Jonghyun ChoiACL Findings 2025 · 2 citations
- White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMsYixin Wan, Kai-Wei ChangACL 2025 · 9 citations
- Mitigating Gender Bias via Fostering Exploratory Thinking in LLMsKangda Wei, Hasnat Md Abdullah, Ruihong HuangEMNLP Findings 2025
