Lune

ACL Findings2023

Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methods

Josip Jukic, Martin Tutek, Jan Snajder

Abstract

A popular approach to unveiling the black box of neural NLP models is to leverage saliency methods, which assign scalar importance scores to each input component. A common practice for evaluating whether an interpretability method is faithful has been to use evaluation-by-agreement -if multiple methods agree on an explanation, its credibility increases. However, recent work has found that saliency methods exhibit weak rank correlations even when applied to the same model instance and advocated for alternative diagnostic methods. In our work, we demonstrate that rank correlation is sensitive to small perturbations when evaluating agreement and argue that Pearson-r could be a better-suited alternative. We further show that regularization techniques that increase faithfulness of attention explanations also increase agreement between saliency methods. By connecting our findings to instance categories based on training dynamics, we show that the agreement of saliency method explanations is very low for easy-to-learn instances. Finally, we connect the improvement in agreement across instance categories to local representation space statistics of instances, paving the way for work on analyzing which intrinsic model properties improve their predisposition to interpretability methods. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification cites Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining
  • "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification cites A Diagnostic Study of Explainability Techniques for Text Classification
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. A Diagnostic Study of Explainability Techniques for Text ClassificationPepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle AugensteinEMNLP 2020 · 158 citations
  2. "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text ClassificationJasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm et al.EMNLP 2022 · 29 citations
  3. Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and RetrainingAndreas Madsen, Nicholas Meade, Vaibhav Adlakha, Siva ReddyEMNLP Findings 2022 · 19 citations
  4. Dataset Cartography: Mapping and Diagnosing Datasets with Training DynamicsSwabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang et al.EMNLP 2020 · 12 citations
  5. Towards Transparent and Explainable Attention ModelsAkash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra et al.ACL 2020 · 11 citations

Related papers

  1. Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model BehaviorAngie W. Boggust, Benjamin Hoover, Arvind Satyanarayan, Hendrik StrobeltCHI 2022 · 51 citations
  2. Evaluating Explanation Methods for Neural Machine TranslationJierui Li, Lemao Liu, Huayang Li, Guanlin Li et al.ACL 2020 · 24 citations
  3. Sanity Checks for Saliency MetricsRichard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram et al.AAAI 2020 · 204 citations
  4. A Comparative Study of Faithfulness Metrics for Model Interpretability MethodsChun Sik Chan, Huanqi Kong, Guanqing LiangACL 2022
  5. On the Faithfulness of Vision Transformer ExplanationsJunyi Wu, Weitai Kang, Hao Tang, Yuan Hong et al.CVPR 2024 · 8 citations
Dusk over the sea between two cliffs drawn in fine vertical lines