EMNLP Findings2022
Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, Siva Reddy
Abstract
To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for making a prediction. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness.
To answer this question, we propose Recursive ROAR, a new faithfulness metric. This works by recursively masking allegedly important tokens and then retraining the model. The principle is that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using relative area-between-curves (RACU), which allows for easy comparison across papers, models, and tasks.
We evaluate 4 different importance measures on 8 different datasets, using both LSTMattention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and taskdependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Faithfulness Measurable Masked Language Models cites A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference
- Faithfulness Measurable Masked Language Models cites "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification
- MultiViz: Towards Visualizing and Understanding Multimodal Models cites Learning to Deceive with Attention-Based Explanations
Cited by top-tier papers10
- "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text ClassificationJasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm et al.EMNLP 2022 · 29 citations
- A Framework for Learning to Request Rich and Contextually Useful Information from HumansKhanh X. Nguyen, Yonatan Bisk, Hal Daumé IIIICML 2022 · 21 citations
- MultiViz: Towards Visualizing and Understanding Multimodal ModelsPaul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain et al.ICLR 2023 · 15 citations
- A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language InferenceKerem Zaman, Yonatan BelinkovEMNLP 2022 · 7 citations
- Faithfulness Measurable Masked Language ModelsAndreas Madsen, Siva Reddy, Sarath ChandarICML 2024 · 6 citations
Builds on4
- Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Peter Hase, Mohit BansalACL 2020 · 216 citations
- Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words?Cansu Sen, Thomas Hartvigsen, Biao Yin, Xiangnan Kong et al.ACL 2020 · 56 citations
- Learning to Deceive with Attention-Based ExplanationsDanish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig et al.ACL 2020 · 17 citations
- On the Interaction of Belief Bias and ExplanationsAna Valeria Gonzalez, Anna Rogers, Anders SøgaardACL Findings 2021 · 10 citations
Related papers
- Incorporating Attribution Importance for Improving Faithfulness MetricsZhixue Zhao, Nikolaos AletrasACL 2023 · 4 citations
- A Comparative Study of Faithfulness Metrics for Model Interpretability MethodsChun Sik Chan, Huanqi Kong, Guanqing LiangACL 2022
- Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMsWenbo Pan, Zhichao Liu, Xianlong Wang, Yu Haining et al.ICML 2026 · 3 citations
- Are self-explanations from Large Language Models faithful?Andreas Madsen, Sarath Chandar, Siva ReddyACL Findings 2024 · 32 citations
- Learning to Faithfully Rationalize by ConstructionSarthak Jain, Sarah Wiegreffe, Yuval Pinter, Byron C. WallaceACL 2020
