ACL Findings2026
Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains
Finn Schmidt, Jan Philip Wahle, Terry Ruas, Bela Gipp
Abstract
Automatic evaluation metrics are central to the development of machine translation systems, yet their robustness under domain shift remains unclear. Most metrics are developed on the Workshop on Machine Translation (WMT) benchmarks, raising concerns about their robustness to unseen domains. Prior studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. To address these biases, we introduce a systematic multi-annotator Cross-Domain Error-Span-Annotation dataset (CD-ESA), comprising 18.8k human error span annotations across three language pairs, where we fix annotators within each language pair and evaluate translations of the same six translation systems across one seen news domain and two unseen technical domains. Using this dataset, we first find that automatic metrics appear surprisingly robust to domain-shifts at the segment level (up to 0.69 agreement), but this robustness largely disappears once we account for human label variation. Averaging annotations increases inter-annotator agreement by up to +0.11. Metrics struggle on the unseen chemical domain compared to humans (interannotator agreement of 0.78-0.83 vs. 0.96). We recommend comparing metric-human agreement against inter-annotator agreement, rather than comparing raw metric-human agreement alone, when evaluating across different domains.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration cites Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
- Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration cites COMET: A Neural Framework for MT Evaluation
Builds on7
- On the Cross-lingual Transferability of Monolingual RepresentationsMikel Artetxe, Sebastian Ruder, Dani YogatamaACL 2020 · 57 citations
- SciRepEval: A Multi-Format Benchmark for Scientific Document RepresentationsAmanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey et al.EMNLP 2023 · 45 citations
- Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation MetricsNitika Mathur, Timothy Baldwin, Trevor CohnACL 2020 · 14 citations
- Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie CalibrationDaniel Deutsch, George F. Foster, Markus FreitagEMNLP 2023 · 14 citations
- COMET: A Neural Framework for MT EvaluationRicardo Rei, Craig Stewart, Ana C. Farinha, Alon LavieEMNLP 2020 · 6 citations
Related papers
- Enhancing Human Evaluation in Machine Translation with Comparative JudgementYixiao Song, Parker Riley, Daniel Deutsch, Markus FreitagACL 2025 · 2 citations
- Quantifying the Impact of Translation Errors on Multilingual LLM EvaluationKlaudia Thellmann, Bernhard Stadler, Michael Färber, Jens LehmannACL 2026
- Assessing Human-Parity in Machine Translation on the Segment LevelYvette Graham, Christian Federmann, Maria Eskevich, Barry HaddowEMNLP Findings 2020 · 13 citations
- Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level SystemsDmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova et al.EMNLP Findings 2025
- Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level AggregationPaulo R. Cavalin, Pedro Henrique Domingues, Claudio S. PinhanezAAAI 2025 · 5 citations
