ACL Findings2023
Counterfactual Probing for the Influence of Affect and Specificity on Intergroup Bias
Venkata Subrahmanyan Govindarajan, David Beaver, Kyle Mahowald, Junyi Jessy Li
Abstract
While existing work on studying bias in NLP focues on negative or pejorative language use, Govindarajan et al. (2023) offer a revised framing of bias in terms of intergroup social context, and its effects on language behavior. In this paper, we investigate if two pragmatic features (specificity and affect) systematically vary in different intergroup contexts -thus connecting this new framing of bias to language output. Preliminary analysis finds modest correlations between specificity and affect of tweets with supervised intergroup relationship (IGR) labels. Counterfactual probing further reveals that while neural models finetuned for predicting IGR reliably use affect in classification, the model's usage of specificity is inconclusive.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on6
- Towards Controllable Biases in Language GenerationEmily Sheng, Kai-Wei Chang, Prem Natarajan, Nanyun PengEMNLP Findings 2020 · 89 citations
- Null It Out: Guarding Protected Attributes by Iterative Nullspace ProjectionShauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton et al.ACL 2020 · 25 citations
- Social Bias Frames: Reasoning about Social and Power Implications of LanguageMaarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky et al.ACL 2020 · 16 citations
- GoEmotions: A Dataset of Fine-Grained EmotionsDorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan S. Cowen et al.ACL 2020 · 16 citations
- Adversarial Concept Erasure in Kernel SpaceShauli Ravfogel, Francisco Vargas, Yoav Goldberg, Ryan CotterellEMNLP 2022 · 11 citations
Related papers
- This prompt is measuring : evaluating bias evaluation in language modelsSeraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir, Su Lin BlodgettACL Findings 2023 · 10 citations
- Reducing Sentiment Bias in Language Models via Counterfactual EvaluationPo-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth et al.EMNLP Findings 2020 · 124 citations
- Do they mean 'us'? Interpreting Referring Expression variation under Intergroup BiasVenkata S. Govindarajan, Matianyu Zang, Kyle Mahowald, David Beaver et al.EMNLP Findings 2024
- Intrinsic Bias Metrics Do Not Correlate with Application BiasSeraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya et al.ACL 2021
- Language (Technology) is Power: A Critical Survey of "Bias" in NLPSu Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna M. WallachACL 2020 · 68 citations
