ACL Findings2024
Are self-explanations from Large Language Models faithful?
Andreas Madsen, Sarath Chandar, Siva Reddy
Abstract
Instruction-tuned Large Language Models (LLMs) excel at many tasks and will even explain their reasoning, so-called selfexplanations. However, convincing and wrong self-explanations can lead to unsupported confidence in LLMs, thus increasing risk. Therefore, it's important to measure if self-explanations truly reflect the model's behavior. Such a measure is called interpretability-faithfulness and is challenging to perform since the ground truth is inaccessible, and many LLMs only have an inference API. To address this, we propose employing self-consistency checks to measure faithfulness. For example, if an LLM says a set of words is important for making a prediction, then it should not be able to make its prediction without these words. While selfconsistency checks are a common approach to faithfulness, they have not previously been successfully applied to LLM self-explanations for counterfactual, feature attribution, and redaction explanations. Our results demonstrate that faithfulness is explanation, model, and taskdependent, showing self-explanations should not be trusted in general. For example, with sentiment classification, counterfactuals are more faithful for Llama2, feature attribution for Mistral, and redaction for Falcon 40B.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
27 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Faithfulness Measurable Masked Language Models cites Explaining NLP Models via Minimal Contrastive Editing (MiCE)
- Faithfulness Measurable Masked Language Models cites "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification
- "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification cites Explaining NLP Models via Minimal Contrastive Editing (MiCE)
Cited by top-tier papers27
- Measuring Chain of Thought Faithfulness by Unlearning Reasoning StepsMartin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, Yonatan BelinkovEMNLP 2025 · 37 citations
- Multi-Level Explanations for Generative Language ModelsLucas Monteiro Paes, Dennis Wei, Hyo Jin Do, Hendrik Strobelt et al.ACL 2025 · 16 citations
- Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in ExplanationsPedro Lobato Ferreira, Wilker Aziz, Ivan TitovICLR 2026 · 12 citations
- Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description FrameworkLaura Kopf, Nils Feldhus, Kirill Bykov, Philine Lou Bommer et al.NeurIPS 2025 · 12 citations
- LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual ExplanationsHarry Mayne, Ryan Othniel Kearns, Yushi Yang, Andrew M. Bean et al.EMNLP 2025 · 9 citations
Builds on5
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Explaining NLP Models via Minimal Contrastive Editing (MiCE)Alexis Ross, Ana Marasovic, Matthew E. PetersACL Findings 2021 · 86 citations
- "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text ClassificationJasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm et al.EMNLP 2022 · 29 citations
- Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-DeploymentZichao Li, Prakhar Sharma, Xing Han Lù, Jackie Chi Kit Cheung et al.ACL Findings 2022 · 14 citations
- Faithfulness Measurable Masked Language ModelsAndreas Madsen, Siva Reddy, Sarath ChandarICML 2024 · 6 citations
Related papers
- On Measuring Faithfulness or Self-consistency of Natural Language ExplanationsLetitia Parcalabescu, Anette FrankACL 2024
- Calibrating Large Language Models with Sample ConsistencyQing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang et al.AAAI 2025 · 72 citations
- Walk the Talk? Measuring the Faithfulness of Large Language Model ExplanationsKatie Matton, Robert Osazuwa Ness, John V. Guttag, Emre KicimanICLR 2025
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language ExplanationsYanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao et al.ICML 2024 · 90 citations
- Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language ModelsWei Jie Yeo, Ranjan Satapathy, Erik CambriaEMNLP 2025 · 2 citations
