B-score: Detecting biases in large language models using response history
An Vo, Mohammad Reza Taesiri, Daeyoung Kim, Anh Totti Nguyen
Abstract
Large language models (LLMs) often exhibit strong biases, e.g., against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and
(3) Objective. Interestingly, LLMs are able to "debias" themselves in a multi-turn conversation in response to questions that seek a Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases in Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e., accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: b-score.github.io.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Language Models are Few-Shot Learners
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs cites Measuring Massive Multitask Language Understanding
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs cites Language Models are Few-Shot Learners
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites Language Models are Few-Shot Learners
- Same or Not? Enhancing Visual Perception in Vision-Language Models cites Measuring Massive Multitask Language Understanding
- Same or Not? Enhancing Visual Perception in Vision-Language Models cites Vision Language Models are Biased
Cited by top-tier papers3
- Vision Language Models are BiasedAn Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Thi Tuong Vy Dang et al.ICLR 2026 · 68 citations
- Same or Not? Enhancing Visual Perception in Vision-Language ModelsDamiano Marsili, Aditya Mehta, Ryan Y. Lin, Georgia GkioxariCVPR 2026 · 5 citations
- D-Models and E-Models: Diversity-Stability Trade-offs in the Sampling Behavior of Large Language ModelsJia Gu, Liang Pang, Huawei Shen, Xueqi ChengWWW 2026
Builds on19
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMsMiao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li et al.ICLR 2024 · 867 citations
- Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun et al.NeurIPS 2021 · 606 citations
Related papers
- Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign PromptsZhaomin Wu, Mingzhe Du, See-Kiong Ng, Bingsheng HeICLR 2026 · 11 citations
- Cognitive Bias in Decision-Making with LLMsJessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian J. McAuley et al.EMNLP Findings 2024 · 62 citations
- Does Question Really Matter? The Attribution of Answer Bias in LLM EvaluationBoxi Cao, Ruotong Pan, Hongyu Lin, Xianpei Han et al.AAAI 2026
- Fairness Testing of Large Language Models in Role-PlayingXinyue Li, Zhenpeng Chen, Jie M. Zhang, Ying Xiao et al.FSE 2026 · 5 citations
- Bias in the Mirror : Are LLMs opinions robust to their own adversarial attacksVirgile Rennard, Christos Xypolopoulos, Michalis VazirgiannisACL 2025 · 8 citations
