Beyond Static Synthetic Noise: Assessing the Robustness of Large Language Models to Natural Context Variation in the Real World

Robustness evaluation in Question Answering (QA) has predominantly relied on synthetic perturbations that poorly capture natural text evolution in real-world settings, a limitation that becomes more pronounced with the widespread deployment of Large Language Models (LLMs) in dynamic, user-facing environments. In this work, we address this gap by proposing a framework for automatically evaluating QA models under naturally occurring textual perturbations, replacing context passages with revised counterparts from Wikipedia edit histories. Through extensive evaluation on SQUAD across diverse encoder architectures, we construct two challenging sets where human performance remains stable, yet state-of-the-art LLMs exhibit significant degradation, with performance drops of up to 28.28%. These robustness gaps further generalise to more complex QA scenarios, such as DROP and HOTPOTQA. To mitigate these errors, we show that robustness to natural perturbations can be improved via adversarial training for encoder-only models and incontext demonstrations of perturbed instances for LLMs, though a more generalisable and effective defense strategy remains an open challenge 1 . Model SQUAD 1.1 SQUAD 2.0 Overall (Ans./Unans.) distilbert-base -0.6 -0.71 (-2.76/1.71) bert-base-cased -0.21 -0.63 (-1.84/0.6) bert-base-uncased -0.87 -0.49 (-1.88/0.94) bert-large-cased -0.63 -0.53 (-1.61/0.55) bert-large-uncased -0.35 -1.38 (-2.51/-0.24) spanbert-base-cased -0.26 -1.24 (-2.66/0.15) spanbert-large-cased -0.51 -1.20 (-1.9/-0.56) roberta-base -0.61 -0.60 (-2.09/0.81) roberta-large -0.29 -1.52 (-2.6/-0.54) albert-base-v1 -1.0 -1.07 (-2.02/-0.22) albert-base-v2 -0.34 -1.08 (-2.03/-0.22) albert-large-v1 -0.42 -0.41 (-1.42/0.52) albert-large-v2 -0.8 -0.69 (-1.66/0.22) albert-xxlarge-v1 -0.75 -1.23 (-3.06/0.49) albert-xxlarge-v2 -0.46 -1.28 (-3.02/0.36)

deberta-large -0.52 -1.05