ACL Findings2024
OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection
Chenyang Huang, Abbas Ghaddar, Ivan Kobyzev, Mehdi Rezagholizadeh, Osmar Zaïane, Boxing Chen
Abstract
Recently, there has been considerable attention on detecting hallucinations and omissions in Machine Translation (MT) systems. The two dominant approaches to tackle this task involve analyzing the MT system's internal states or relying on the output of external tools, such as sentence similarity or MT quality estimators. In this work, we introduce OTTAWA, a novel Optimal Transport (OT)-based word aligner specifically designed to enhance the detection of hallucinations and omissions in MT systems. Our approach explicitly models the missing alignments by introducing a "null" vector, for which we propose a novel one-side constrained OT setting to allow an adaptive null alignment. Our approach yields competitive results compared to state-of-the-art methods across 18 language pairs on the HalOmi benchmark. In addition, it shows promising features, such as the ability to distinguish between both error types and perform word-level detection without accessing the MT system's internal states. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages cites Unsupervised Cross-lingual Representation Learning at Scale
- SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings cites A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
- SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings cites Unsupervised Cross-lingual Representation Learning at Scale
- Detecting Hallucinated Content in Conditional Neural Sequence Generation cites SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings
- Detecting Hallucinated Content in Conditional Neural Sequence Generation cites Unsupervised Cross-lingual Representation Learning at Scale
Builds on19
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Detecting Hallucinated Content in Conditional Neural Sequence GenerationChunting Zhou, Graham Neubig, Jiatao Gu, Mona T. Diab et al.ACL Findings 2021 · 122 citations
- Bitext Mining Using Distilled Sentence Representations for Low-Resource LanguagesKevin Heffernan, Onur Çelebi, Holger SchwenkEMNLP Findings 2022 · 47 citations
- SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized EmbeddingsMasoud Jalili Sabet, Philipp Dufter, François Yvon, Hinrich SchützeEMNLP Findings 2020 · 45 citations
- A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERTMasaaki Nagata, Katsuki Chousa, Masaaki NishinoEMNLP 2020 · 38 citations
Related papers
- Optimal Transport for Unsupervised Hallucination Detection in Neural Machine TranslationNuno Miguel Guerreiro, Pierre Colombo, Pablo Piantanida, André F. T. MartinsACL 2023 · 8 citations
- Word Alignment as Preference for Machine TranslationQiyu Wu, Masaaki Nagata, Zhongtao Miao, Yoshimasa TsuruokaEMNLP 2024 · 4 citations
- HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine TranslationDavid Dale, Elena Voita, Janice Lam, Prangthip Hansanti et al.EMNLP 2023 · 8 citations
- Prevent the Language Model from being Overconfident in Neural Machine TranslationMengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou et al.ACL 2021
- Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language ModelsKenza Benkirane, Laura Gongas, Shahar Pelles, Naomi Fuchs et al.EMNLP Findings 2024 · 7 citations
