Lune

ACL Findings2024

Investigating the Impact of Model Instability on Explanations and Uncertainty

Sara Marjanovic, Isabelle Augenstein, Christina Lioma

Abstract

Explainable AI methods facilitate the understanding of model behaviour, yet, small, imperceptible perturbations to inputs can vastly distort explanations. As these explanations are typically evaluated holistically, before model deployment, it is difficult to assess when a particular explanation is trustworthy. Some studies have tried to create confidence estimators for explanations, but none have investigated an existing link between uncertainty and explanation quality. We artificially simulate epistemic uncertainty in text input by introducing noise at inference time. In this large-scale empirical study, we insert different levels of noise perturbations and measure the effect on the output of pre-trained language models and different uncertainty metrics. Realistic perturbations have minimal effect on performance and explanations, yet masking has a drastic effect. We find that high uncertainty doesn't necessarily imply low explanation plausibility; the correlation between the two metrics can be moderately positive when noise is exposed during the training process. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain. Furthermore, when predictive and epistemic uncertainty measures are over-confident, the robustness of a saliency map to perturbation can indicate model stability issues. Integrated Gradients shows the overall greatest robustness to perturbation, while still showing model-specific patterns in performance; however, this phenomenon is limited to smaller Transformer-based language models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BERT & Family Eat Word Salad: Experiments with Text Understanding cites Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
  2. Reliable Post hoc Explanations: Modeling Uncertainty in ExplainabilityDylan Slack, Anna Hilgard, Sameer Singh, Himabindu LakkarajuNeurIPS 2021 · 240 citations
  3. A Diagnostic Study of Explainability Techniques for Text ClassificationPepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle AugensteinEMNLP 2020 · 158 citations
  4. BERT & Family Eat Word Salad: Experiments with Text UnderstandingAshim Gupta, Giorgi Kvernadze, Vivek SrikumarAAAI 2021 · 77 citations
  5. Evaluating the Robustness of Neural Language Models to Input PerturbationsMilad Moradi, Matthias SamwaldEMNLP 2021 · 64 citations

Related papers

  1. Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question AnsweringYangyi Li, Mengdi HuaiEMNLP Findings 2025 · 2 citations
  2. Distinguishing the Knowable from the Unknowable with Language ModelsGustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak et al.ICML 2024 · 44 citations
  3. Decomposing Uncertainty for Large Language Models through Input Clarification EnsemblingBairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas et al.ICML 2024 · 113 citations
  4. Interpreting the Robustness of Neural NLP Models to Textual PerturbationsYunxiang Zhang, Liangming Pan, Samson Tan, Min-Yen KanACL Findings 2022 · 13 citations
  5. Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-AnsweringYavuz Faruk Bakman, Sungmin Kang, Zhiqi Huang, Duygu Nur Yaldiz et al.ICLR 2026 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines