EMNLP Findings2023
Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition
Isaac Slaughter, Craig Greenberg, Reva Schwartz, Aylin Caliskan
Abstract
Previous work has established that a person's demographics and speech style affect how well speech processing models perform for them. But where does this bias come from? In this work, we present the Speech Embedding Association Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is inspired by word embedding association tests in natural language processing, which quantify intrinsic bias in a model's representations of different concepts, such as race or valencesomething's pleasantness or unpleasantnessand capture the extent to which a model trained on large-scale socio-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech models (including 4 models also trained on multilingual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European-Americans over African-Americans, with females over males, with U.S. accented speakers over non-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models contain these biases, we also show that they can have real world effects. We compare biases found in pre-trained models to biases in downstream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69%), the group that is more associated with positive valence as indicated by the SpEAT also tends to be predicted as speaking with higher valence by the downstream model. Our work provides evidence that, like text and image-based models, pretrained speech based-models frequently learn human-like biases when trained on large-scale socio-cultural datasets. Our work also shows that bias found in pre-trained models can propagate to the downstream task of SER.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- "It feels like we're not meeting the criteria": Examining and Mitigating the Cascading Effects of Bias in Automatic Speech Recognition in Spoken Language Interfaces cites Robust Speech Recognition via Large-Scale Weak Supervision
- Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models cites TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
- Robust Speech Recognition via Large-Scale Weak Supervision cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content Evaluations cites "It feels like we're not meeting the criteria": Examining and Mitigating the Cascading Effects of Bias in Automatic Speech Recognition in Spoken Language Interfaces
Cited by top-tier papers2
- "It feels like we're not meeting the criteria": Examining and Mitigating the Cascading Effects of Bias in Automatic Speech Recognition in Spoken Language InterfacesKelechi Ezema, Chelsea Chandler, Rosy Southwell, Niranjan Cholendiran et al.CHI 2025 · 8 citations
- Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content EvaluationsKowe Kadoma, Priyal Shrivastava, Mor NaamanCHI 2026 · 1 citation
Builds on8
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet ClassificationFrancesco Barbieri, José Camacho-Collados, Luis Espinosa Anke, Leonardo NevesEMNLP Findings 2020 · 720 citations
- Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language ModelsRyan Steed, Swetasudha Panda, Ari Kobren, Michael L. WickACL 2022 · 52 citations
- Towards Fairness in Visual Recognition: Effective Strategies for Bias MitigationZeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova et al.CVPR 2020
Related papers
- The MERSA Dataset and a Transformer-Based Approach for Speech Emotion RecognitionEnshi Zhang, Rafael Trujillo, Christian PoellabauerACL 2024
- ValNorm Quantifies Semantics to Reveal Consistent Valence Biases Across Languages and Over CenturiesAutumn Toney, Aylin CaliskanEMNLP 2021
- Mind Your Bias: A Critical Review of Bias Detection Methods for Contextual Language ModelsSilke Husse, Andreas SpitzEMNLP Findings 2022 · 9 citations
- Do self-supervised speech models develop human-like perception biases?Juliette Millet, Ewan DunbarACL 2022 · 27 citations
- The (Undesired) Attenuation of Human Biases by MultilingualityCristina España-Bonet, Alberto Barrón-CedeñoEMNLP 2022 · 5 citations
