Lune

EMNLP Findings2025

Social Bias Evaluation for Large Language Models Requires Prompt Variations

Rem Hida, Masahiro Kaneko, Naoaki Okazaki

Abstract

Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mitigate these biases accurately. Previous studies use downstream tasks to examine the degree of social biases for evaluation and mitigation. While the output of LLMs highly depends on prompts, prior works evaluating and mitigating bias have often relied on a limited variety of prompts. In this paper, we investigate the sensitivity of LLMs when changing prompt variations (task instruction, few-shot examples, debias-prompt) by analyzing task performance and social bias of LLMs. Our experimental results reveal that LLM rankings fluctuate across prompts for both task performance and social bias. We also confirmed that the impact of format changes can differ for each bias category. Performance improvement from prompt settings may not result in reduced bias. Moreover, the ambiguity of instances is a common factor in LLM sensitivity to prompts across advanced LLMs. We recommend using diverse prompts, as in this study, to compare the effects of prompts on social bias in LLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • A Multilingual Social Bias Benchmark Incorporating Thinking Processes cites Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  • A Multilingual Social Bias Benchmark Incorporating Thinking Processes cites Language Models are Few-Shot Learners
  • Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
  • Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Language Models are Few-Shot Learners
  • Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) cites Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
  • Fairness Mediator: Neutralize Stereotype Associations to Mitigate Bias in Large Language Models cites Measuring Massive Multitask Language Understanding
  • Fairness Mediator: Neutralize Stereotype Associations to Mitigate Bias in Large Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias DetectorHaoyan Yang, Runxue Bao, Cao (Danica) Xiao, Jun Ma et al.NeurIPS 2025 · 15 citations
  2. Fairness Mediator: Neutralize Stereotype Associations to Mitigate Bias in Large Language ModelsYisong Xiao, Aishan Liu, Siyuan Liang, Xianglong Liu et al.ISSTA 2025 · 2 citations
  3. A Multilingual Social Bias Benchmark Incorporating Thinking ProcessesMasahiro Kaneko, Danushka Bollegala, Timothy BaldwinACL 2026
  4. Token-Aware Editing of Internal Activations for Large Language Model AlignmentTianbo Wang, Yuqing Ma, Kewei Liao, Chengzhao Yang et al.EMNLP 2025
  5. Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)Leander Girrbach, Stephan Alaniz, Yiran Huang, Trevor Darrell et al.ICLR 2025

Builds on30

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations
  4. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
  5. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingMelanie Sclar, Yejin Choi, Yulia Tsvetkov, Alane SuhrICLR 2024 · 682 citations

Related papers

  1. Bi-directional Bias Attribution: Debiasing Large Language Models without Modifying PromptsYujie Lin, Kunquan Li, Yixuan Liao, Xiaoxin Chen et al.ICLR 2026 · 6 citations
  2. POSIX: A Prompt Sensitivity Index For Large Language ModelsAnwoy Chatterjee, H. S. V. N. S. Kowndinya Renduchintala, Sumit Bhatia, Tanmoy ChakrabortyEMNLP Findings 2024 · 36 citations
  3. Prompting Fairness: Integrating Causality to Debias Large Language ModelsJingling Li, Zeyu Tang, Xiaoyu Liu, Peter Spirtes et al.ICLR 2025
  4. Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender BiasesBufan Gao, Elisa KreissEMNLP 2025 · 1 citation
  5. Harnessing Non-Adversarial Robustness in Large Language ModelsQinghua Zhou, Ellina Aleshina, Andrey Lovyagin, Oleg Somov et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines