EMNLP Findings2020
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?
Peter Hase, Shiyue Zhang, Harry Xie, Mohit Bansal
Abstract
Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
37 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- NILE : Natural Language Inference with Faithful Natural Language Explanations cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting cites Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations cites Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations cites NILE : Natural Language Inference with Faithful Natural Language Explanations
- Knowledge-Grounded Self-Rationalization via Extractive and Natural Language Explanations cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
- Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations cites Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
- Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
- Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations cites NILE : Natural Language Inference with Faithful Natural Language Explanations
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations cites Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
- ERASER: A Benchmark to Evaluate Rationalized NLP Models cites Learning to Faithfully Rationalize by Construction
- Learning to Faithfully Rationalize by Construction cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
Cited by top-tier papers37
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language ExplanationsYanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao et al.ICML 2024 · 90 citations
- Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model ExplanationsSiddhant Arora, Danish Pruthi, Norman M. Sadeh, William W. Cohen et al.AAAI 2022 · 47 citations
- Knowledge-Grounded Self-Rationalization via Extractive and Natural Language ExplanationsBodhisattwa Prasad Majumder, Oana Camburu, Thomas Lukasiewicz, Julian J. McAuleyICML 2022 · 40 citations
- SCOTT: Self-Consistent Chain-of-Thought DistillationPeifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao et al.ACL 2023 · 39 citations
Builds on5
- Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Peter Hase, Mohit BansalACL 2020 · 216 citations
- ERASER: A Benchmark to Evaluate Rationalized NLP ModelsJay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric P. Lehman et al.ACL 2020 · 36 citations
- NILE : Natural Language Inference with Faithful Natural Language ExplanationsSawan Kumar, Partha P. TalukdarACL 2020 · 15 citations
- Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language LearningAngeliki Lazaridou, Anna Potapenko, Olivier TielemanACL 2020 · 11 citations
- Learning to Faithfully Rationalize by ConstructionSarthak Jain, Sarah Wiegreffe, Yuval Pinter, Byron C. WallaceACL 2020
Related papers
- Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language ExplanationsBingsheng Yao, Prithviraj Sen, Lucian Popa, James A. Hendler et al.ACL 2023 · 4 citations
- ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated SimulatabilityAntonin Poché, Alon Jacovi, Agustin Martin Picard, Victor Boutin et al.ACL 2025 · 8 citations
- Directly Optimizing Natural Language Explanations for Behavioral Faithfulness: Simulatability and RecoverabilityAdvaith Malladi, Shashank SrivastavaICML 2026
- Explain the Synth: Interpretable Evaluation of LLM Data SynthesisYue Yang, Fan Yang, Yu Bai, Hao WangACL 2026
- Logical Satisfiability of Counterfactuals for Faithful Explanations in NLISuzanna Sia, Anton Belyy, Amjad Almahairi, Madian Khabsa et al.AAAI 2023 · 17 citations
