Lune

EMNLP Findings2023

Statistically Profiling Biases in Natural Language Reasoning Datasets and Models

Shanshan Huang, Kenny Q. Zhu

Abstract

Recent studies have shown that many natural language understanding and reasoning datasets contain statistical cues that can be exploited by NLP models, resulting in an overestimation of their capabilities. Existing methods, such as "hypothesis-only" tests and CheckList, are limited in identifying these cues and evaluating model weaknesses. We introduce ICQ (I-See-Cue), a lightweight, general statistical profiling framework that automatically identifies potential biases in multiple-choice NLU datasets without requiring additional test cases. ICQ assesses the extent to which models exploit these biases through black-box testing, addressing the limitations of current methods. In this work, we conduct a comprehensive evaluation of statistical biases in 10 popular NLU datasets and 4 models, confirming prior findings, revealing new insights, and offering an online demonstration system to encourage users to assess their own datasets and models. Furthermore, we present a case study on investigating ChatGPT's bias, providing valuable recommendations for practical applications.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Language Models are Zero-Shot Reasoners cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen et al.EMNLP 2023 · 449 citations
  4. ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningWeihao Yu, Zihang Jiang, Yanfei Dong, Jiashi FengICLR 2020 · 325 citations
  5. Adversarial Filters of Dataset BiasesRonan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers et al.ICML 2020 · 242 citations

Related papers

  1. Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM ReasoningXuyang Wu, Jinming Nian, Ting-Ruen Wei, Zhiqiang Tao et al.EMNLP Findings 2025 · 5 citations
  2. Biases in the Blind Spot: Detecting What LLMs Fail to MentionIván Arcuschin, David Chanin, Adrià Garriga-Alonso, Oana-Maria CamburuICML 2026 · 7 citations
  3. Justice or Prejudice? Quantifying Biases in LLM-as-a-JudgeJiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen et al.ICLR 2025
  4. BBQ: A hand-built bias benchmark for question answeringAlicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar et al.ACL Findings 2022 · 186 citations
  5. CausalityCheck: A Framework for Evaluating Causal Reasoning in Large Language ModelsJiang Li, Zehua Duo, Guanglai Gao, Xiangdong SuACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines