EMNLP Findings2025
Neutral Is Not Unbiased: Evaluating Implicit and Intersectional Identity Bias in LLMs Through Structured Narrative Scenarios
Saba Ghanbari Haez, Mauro Dragoni
摘要
Large Language Models often reproduce societal biases, yet most evaluations overlook how such biases evolve across nuanced contexts or intersecting identities. We introduce a scenario-based evaluation framework built on 100 narrative tasks, designed to be neutral at baseline and systematically modified with gender and age cues. Grounded in the theory of Normative-Narrative Scenarios, our approach provides ethically coherent and socially plausible settings for probing model behavior. Analyzing responses from five leading LLMs-GPT-4o, LLaMA 3.1, Qwen2.5, Phi-4, and Mistral-using Critical Discourse Analysis and quantitative linguistic metrics, we find consistent evidence of bias. Gender emerges as the dominant axis of bias, with intersectional cues (e.g., age and gender combined) further intensifying disparities. Our results underscore the value of dynamic narrative progression for detecting implicit, systemic biases in Large Language Models.
问 Lune
尽管问,回答会引用原文。
Lune 会读完全文和相关的 Paper,每个回答都附上所依据的原文段落。
它引用的 Paper11
- Towards Controllable Biases in Language GenerationEmily Sheng, Kai-Wei Chang, Prem Natarajan, Nanyun PengEMNLP Findings 2020 · 89 次引用
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative ModelsMahammed Kamruzzaman, Md. Minul Islam Shovon, Gene Louis KimACL Findings 2024 · 18 次引用
- Queens are Powerful too: Mitigating Gender Bias in Dialogue GenerationEmily Dinan, Angela Fan, Adina Williams, Jack Urbanek 等EMNLP 2020 · 14 次引用
- Intersectional Stereotypes in Large Language Models: Dataset and AnalysisWeicheng Ma, Brian Chiang, Tong Wu, Lili Wang 等EMNLP Findings 2023 · 7 次引用
- Gender Bias in Decision-Making with Large Language Models: A Study of Relationship ConflictsSharon Levy, William D. Adler, Tahilin Sanchez Karver, Mark Dredze 等EMNLP Findings 2024 · 7 次引用
相关 Paper
- Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted TextTunazzina IslamACL Findings 2026
- Evaluating Gender Bias of LLMs in Making Morality JudgementsDivij Bajaj, Yuanyuan Lei, Jonathan Tong, Ruihong HuangEMNLP Findings 2024 · 3 次引用
- Large Language Models Still Exhibit Bias in Long TextWonje Jeung, Dongjae Jeon, Ashkan Yousefpour, Jonghyun ChoiACL Findings 2025 · 2 次引用
- White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMsYixin Wan, Kai-Wei ChangACL 2025 · 9 次引用
- Mitigating Gender Bias via Fostering Exploratory Thinking in LLMsKangda Wei, Hasnat Md Abdullah, Ruihong HuangEMNLP Findings 2025
