Lune

ACL Findings2023

Stubborn Lexical Bias in Data and Models

Sofia Serrano, Jesse Dodge, Noah A. Smith

Abstract

In NLP, recent work has seen increased focus on spurious correlations between various features and labels in training data, and how these influence model behavior. However, the presence and effect of such correlations are typically examined feature by feature. We investigate the cumulative impact on a model of many such intersecting features. Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data. We select two tasks-natural language inference and duplicate-question detection-for which any unigram feature on its own should ideally be uninformative, which gives us a large pool of automatically extracted features with which to experiment. The large size of this pool allows us to investigate the intersection of features spuriously associated with (potentially different) labels. We then apply an optimization approach to reweight the training data, reducing thousands of spurious correlations, and examine how doing so affects models trained on the reweighted data. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models, including worsened bias for slightly more complex features (bigrams). We close with discussion about the implications of our results on what it means to "debias" training data, and how issues of data quality can affect model bias.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasets cites End-to-End Bias Mitigation by Modelling Biases in Corpora
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Are Models Biased on Text without Gender-related Language?Catarina G. Belém, Preethi Seshadri, Yasaman Razeghi, Sameer SinghICLR 2024 · 16 citations
  2. Resolving Lexical Bias in Model EditingHammad Rizwan, Domenic Rosati, Ga Wu, Hassan SajjadICML 2025

Builds on8

  1. Evaluating Models' Local Decision Boundaries via Contrast SetsMatt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant et al.EMNLP Findings 2020 · 260 citations
  2. End-to-End Bias Mitigation by Modelling Biases in CorporaRabeeh Karimi Mahabadi, Yonatan Belinkov, James HendersonACL 2020 · 136 citations
  3. A Closer Look at How Fine-tuning Changes BERTYichu Zhou, Vivek SrikumarACL 2022 · 84 citations
  4. Generating Data to Mitigate Spurious Correlations in Natural Language Inference DatasetsYuxiang Wu, Matt Gardner, Pontus Stenetorp, Pradeep DasigiACL 2022 · 74 citations
  5. Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language ModelsRyan Steed, Swetasudha Panda, Ari Kobren, Michael L. WickACL 2022 · 52 citations

Related papers

  1. Are All Spurious Features in Natural Language Alike? An Analysis through a Causal LensNitish Joshi, Xiang Pan, He HeEMNLP 2022 · 19 citations
  2. Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
  3. Competency Problems: On Finding and Removing Artifacts in Language DataMatt Gardner, William Merrill, Jesse Dodge, Matthew E. Peters et al.EMNLP 2021 · 72 citations
  4. Towards Robustifying NLI Models Against Lexical Dataset BiasesXiang Zhou, Mohit BansalACL 2020 · 36 citations
  5. Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of SamplesWeiwei Li, Junzhuo Liu, Yuanyuan Ren, Yuchen Zheng et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines