Lune

EMNLP Findings2023

Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks

Yichen Huang, Timothy Baldwin

Abstract

We investigate MT evaluation metric performance on adversarially-synthesized texts, to shed light on metric robustness. We experiment with word-and character-level attacks on three popular machine translation metrics: BERTScore, BLEURT, and COMET. Our human experiments validate that automatic metrics tend to overpenalize adversarially-degraded translations. We also identify inconsistencies in BERTScore ratings, where it judges the original sentence and the adversarially-degraded one as similar, while judging the degraded translation as notably worse than the original with respect to the reference. We identify patterns of brittleness that motivate more robust metric development.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BLEURT: Learning Robust Metrics for Text Generation cites BERTScore: Evaluating Text Generation with BERT
  • Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites BLEURT: Learning Robust Metrics for Text Generation
  • Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages cites BERTScore: Evaluating Text Generation with BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian LanguagesAmir Hossein Yari, Kalmit Kulkarni, Ahmad Raza Khan, Fajri KotoACL 2026

Builds on9

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. Tailor: Generating and Perturbing Text with Semantic ControlsAlexis Ross, Tongshuang Wu, Hao Peng, Matthew E. Peters et al.ACL 2022 · 85 citations
  3. Making Monolingual Sentence Embeddings Multilingual using Knowledge DistillationNils Reimers, Iryna GurevychEMNLP 2020 · 54 citations
  4. BLEURT: Learning Robust Metrics for Text GenerationThibault Sellam, Dipanjan Das, Ankur P. ParikhACL 2020 · 40 citations
  5. Multi-granularity Textual Adversarial Attack with Behavior CloningYangyi Chen, Jin Su, Wei WeiEMNLP 2021 · 29 citations

Related papers

  1. DEMETR: Diagnosing Evaluation Metrics for TranslationMarzena Karpinska, Nishant Raj, Katherine Thai, Yixiao Song et al.EMNLP 2022 · 18 citations
  2. BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk TrainingYiming Yan, Tao Wang, Chengqi Zhao, Shujian Huang et al.ACL 2023 · 7 citations
  3. Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level AggregationPaulo R. Cavalin, Pedro Henrique Domingues, Claudio S. PinhanezAAAI 2025 · 5 citations
  4. On the Blind Spots of Model-Based Evaluation Metrics for Text GenerationTianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar et al.ACL 2023 · 10 citations
  5. Extrinsic Evaluation of Machine Translation MetricsNikita Moghe, Tom Sherborne, Mark Steedman, Alexandra BirchACL 2023 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines