Lune

ACL Findings2025

A Representation Level Analysis of NMT Model Robustness to Grammatical Errors

Abderrahmane Issam, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis

Abstract

Understanding robustness is essential for building reliable NLP systems. Unfortunately, in the context of machine translation, previous work mainly focused on documenting robustness failures or improving robustness. In contrast, we study robustness from a model representation perspective by looking at internal model representations of ungrammatical inputs and how they evolve through model layers. For this purpose, we perform Grammatical Error Detection (GED) probing and representational similarity analysis. Our findings indicate that the encoder first detects the grammatical error, then corrects it by moving its representation toward the correct form. To understand what contributes to this process, we turn to the attention mechanism where we identify what we term Robustness Heads. We find that Robustness Heads attend to interpretable linguistic units when responding to grammatical errors, and that when we fine-tune models for robustness, they tend to rely more on Robustness Heads for updating the ungrammatical word representation. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. It's Morphin' Time! Combating Linguistic Discrimination with Inflectional PerturbationsSamson Tan, Shafiq R. Joty, Min-Yen Kan, Richard SocherACL 2020 · 88 citations
  2. A Closer Look at How Fine-tuning Changes BERTYichu Zhou, Vivek SrikumarACL 2022 · 84 citations
  3. How transfer learning impacts linguistic knowledge in deep NLP models?Nadir Durrani, Hassan Sajjad, Fahim DalviACL Findings 2021 · 39 citations
  4. Revisiting Robust Neural Machine Translation: A Transformer Case StudyPeyman Passban, Puneeth S. M. Saladi, Qun LiuEMNLP Findings 2021 · 8 citations
  5. Robustness-Aware Word Embedding Improves Certified Robustness to Adversarial Word SubstitutionsYibin Wang, Yichen Yang, Di He, Kun HeACL Findings 2023 · 8 citations

Related papers

  1. How does BERT's attention change when you fine-tune? An analysis methodology and a case study in negation scopeYiyun Zhao, Steven BethardACL 2020 · 35 citations
  2. The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate ItLeonardo Bertolazzi, Philipp Mondorf, Barbara Plank, Raffaella BernardiEMNLP 2025 · 8 citations
  3. Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model PredictionsJavier Ferrando, Marta R. Costa-jussàEMNLP Findings 2021 · 14 citations
  4. An Evaluation Resource for Grounding Translation ErrorsSujin Chen, Kang Wang, Zixuan Zhou, Xiangyu Duan et al.EMNLP Findings 2025
  5. Debiasing Methods in Natural Language Understanding Make Bias More AccessibleMichael Mendelson, Yonatan BelinkovEMNLP 2021 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines