EMNLP Findings2023
Toxicity in Multilingual Machine Translation at Scale
Marta R. Costa-jussà, Eric Michael Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, Carlos Escolano
Abstract
Machine Translation systems can produce different types of errors, some of which are characterized as critical or catastrophic due to the specific negative impact that they can have on users.In this paper we focus on one type of critical error: added toxicity.We evaluate and analyze added toxicity when translating a large evaluation dataset (HOLISTICBIAS, over 472k sentences, covering 13 demographic axes) from English into 164 languages.An automatic toxicity evaluation shows that added toxicity across languages varies from 0% to 5%.The output languages with the most added toxicity tend to be low-resource ones, and the demographic axes with the most added toxicity include sexual orientation, gender and sex, and ability.We also perform human evaluation on a subset of 8 translation directions, confirming the prevalence of true added toxicity.We use a measurement of the amount of source contribution to the translation, where a low source contribution implies hallucination, to interpret what causes toxicity.Making use of the input attributions allows us to explain toxicity, because the source contributions significantly correlate with toxicity for 84% of languages studied.Given our findings, our recommendations to reduce added toxicity are to curate training data to avoid mistranslations, mitigate hallucination and check unstable translations.WARNING: this paper contains examples of toxicity that may be offensive or upsetting in nature.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Prompting PaLM for Translation: Assessing Strategies and Performance cites Language Models are Few-Shot Learners
- Perturbation Augmentation for Fairer NLP cites Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
Cited by top-tier papers3
- Prompting PaLM for Translation: Assessing Strategies and PerformanceDavid Vilar, Markus Freitag, Colin Cherry, Jiaming Luo et al.ACL 2023 · 70 citations
- The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating ItZheng Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen H. Bach et al.EMNLP 2025 · 2 citations
- What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered StudyBeatrice Savoldi, Sara Papi, Matteo Negri, Ana Guerberof Arenas et al.EMNLP 2024 · 1 citation
Builds on8
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Attention is Not Only a Weight: Analyzing Transformers with Vector NormsGoro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro InuiEMNLP 2020 · 138 citations
- Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation BenchmarkingZhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain et al.NeurIPS 2021 · 76 citations
- Accurate Word Alignment Induction from Neural Machine TranslationYun Chen, Yang Liu, Guanhua Chen, Xin Jiang et al.EMNLP 2020 · 56 citations
- Perturbation Augmentation for Fairer NLPRebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith et al.EMNLP 2022 · 54 citations
Related papers
- Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at ScaleMarta R. Costa-jussà, Pierre Andrews, Eric Michael Smith, Prangthip Hansanti et al.EMNLP 2023 · 2 citations
- HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine TranslationDavid Dale, Elena Voita, Janice Lam, Prangthip Hansanti et al.EMNLP 2023 · 8 citations
- "I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor DatasetEric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani et al.EMNLP 2022 · 56 citations
- HAT: Hallucination Annotation for TranslationRajen Chatterjee, Xintong Li, Paisarn Charoenpornsawat, Allen LeeACL 2026
- A Survey of Toxicity Mitigation Strategies for Multilingual Language ModelsSoham Dan, Himanshu Beniwal, Thomas HartvigsenACL Findings 2026
