BiasAsker: Measuring the Bias in Conversational AI System
Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, Michael R. Lyu
Abstract
Powered by advanced Artificial Intelligence (AI) techniques, conversational AI systems, such as ChatGPT and digital assistants like Siri, have been widely deployed in daily life. However, such systems may still produce content containing biases and stereotypes, causing potential social problems. Due to the data-driven, black-box nature of modern AI techniques, comprehensively identifying and measuring biases in conversational systems remains a challenging task. Particularly, it is hard to generate inputs that can comprehensively trigger potential bias due to the lack of data containing both social groups as well as biased properties. In addition, modern conversational systems can produce diverse responses (e.g., chatting and explanation), which makes existing bias detection methods simply based on the sentiment and the toxicity hardly being adopted. In this paper, we propose BiasAsker, an automated framework to identify and measure social bias in conversational AI systems. To obtain social groups and biased properties, we construct a comprehensive social bias dataset, containing a total of 841 groups and 8,110 biased properties. Given the dataset, BiasAsker automatically generates questions and adopts a novel method based on existence measurement to identify two types of biases (i.e., absolute bias and related bias) in conversational systems. Extensive experiments on 8 commercial systems and 2 famous research models, such as Chat-GPT and GPT-3, show that 32.83% of the questions generated by BiasAsker can trigger biased behaviors in these widely deployed conversational systems. All the code, data, and experimental results have been released to facilitate future research.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection cites An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software
- An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software cites RobOT: Robustness-Oriented Testing for Deep Learning Systems
- An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software cites Improving Adversarial Transferability via Neuron Attribution-based Attacks
- An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software cites Hidden Voice Commands
Cited by top-tier papers16
- Measuring Political Bias in Large Language Models: What Is Said and How It Is SaidYejin Bang, Delong Chen, Nayeon Lee, Pascale FungACL 2024 · 21 citations
- VisBias: Measuring Explicit and Implicit Social Biases in Vision Language ModelsJen-Tse Huang, Jiantong Qin, Jianping Zhang, Youliang Yuan et al.EMNLP 2025 · 13 citations
- Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective DetectionYuxi Li, Yi Liu, Gelei Deng, Ying Zhang et al.FSE 2024 · 12 citations
- An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation SoftwareWenxuan Wang, Jingyuan Huang, Jen-tse Huang, Chang Chen et al.ASE 2023 · 7 citations
- VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic ManipulationZhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang et al.FSE 2025 · 6 citations
Builds on18
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsSamuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi et al.EMNLP Findings 2020 · 692 citations
- Hidden Voice CommandsNicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang et al.USENIX Security 2016 · 672 citations
- Improving Adversarial Transferability via Neuron Attribution-based AttacksJianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang et al.CVPR 2022 · 140 citations
- DeepCrime: mutation testing of deep learning systems based on real faultsNargiz Humbatova, Gunel Jahangirova, Paolo TonellaISSTA 2021 · 114 citations
- RobOT: Robustness-Oriented Testing for Deep Learning SystemsJingyi Wang, Jialuo Chen, Youcheng Sun, Xingjun Ma et al.ICSE 2021 · 62 citations
Related papers
- Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue SystemsYixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng et al.EMNLP Findings 2023 · 8 citations
- Uncovering and Quantifying Social Biases in Code GenerationYan Liu, Xiaokang Chen, Yan Gao, Zhe Su et al.NeurIPS 2023 · 47 citations
- Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM ReasoningXuyang Wu, Jinming Nian, Ting-Ruen Wei, Zhiqiang Tao et al.EMNLP Findings 2025 · 5 citations
- Cognitive Bias in Decision-Making with LLMsJessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian J. McAuley et al.EMNLP Findings 2024 · 62 citations
- RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language ModelsSoumya Barikeri, Anne Lauscher, Ivan Vulic, Goran GlavasACL 2021
