ACL Findings2023
Sequential Integrated Gradients: a simple but effective method for explaining language models
Joseph Enguehard
Abstract
Several explanation methods such as Integrated Gradients (IG) can be characterised as pathbased methods, as they rely on a straight line between the data and an uninformative baseline. However, when applied to language models, these methods produce a path for each word of a sentence simultaneously, which could lead to creating sentences from interpolated words either having no clear meaning, or having a significantly different meaning compared to the original sentence. In order to keep the meaning of these sentences as close as possible to the original one, we propose Sequential Integrated Gradients (SIG), which computes the importance of each word in a sentence by keeping fixed every other words, only creating interpolations between the baseline and the word of interest. Moreover, inspired by the training procedure of several language models, we also propose to replace the baseline token "pad" with the trained token "mask". While being a simple improvement over the original IG method, we show on various models and datasets that SIG proves to be a very effective method for explaining language models. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
14 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations cites Discretized Integrated Gradients for Explaining Language Models
- Data-faithful Feature Attribution: Mitigating Unobservable Confounders via Instrumental Variables cites A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions
- Discretized Integrated Gradients for Explaining Language Models cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
Cited by top-tier papers14
- ContextCite: Attributing Model Generation to ContextBenjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, Aleksander MadryNeurIPS 2024 · 118 citations
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence AssociationsYiyou Sun, Yu Gai, Lijie Chen, Abhilasha Ravichander et al.NeurIPS 2025 · 20 citations
- Adversarial Representation Engineering: A General Model Editing Framework for Large Language ModelsYihao Zhang, Zeming Wei, Jun Sun, Meng SunNeurIPS 2024 · 16 citations
- Do LLMs Encode Functional Importance of Reasoning Tokens ?Janvijay Singh, Dilek Hakkani-TürACL 2026 · 3 citations
- Data-faithful Feature Attribution: Mitigating Unobservable Confounders via Instrumental VariablesQiheng Sun, Haocheng Xia, Jinfei LiuNeurIPS 2024 · 3 citations
Builds on5
- Incorporating BERT into Neural Machine TranslationJinhua Zhu, Yingce Xia, Lijun Wu, Di He et al.ICLR 2020 · 391 citations
- Counterfactual Explanations Can Be ManipulatedDylan Slack, Anna Hilgard, Himabindu Lakkaraju, Sameer SinghNeurIPS 2021 · 182 citations
- A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron AttributionsDaniel Lundström, Tianjian Huang, Meisam RazaviyaynICML 2022 · 85 citations
- ERASER: A Benchmark to Evaluate Rationalized NLP ModelsJay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric P. Lehman et al.ACL 2020 · 36 citations
- Discretized Integrated Gradients for Explaining Language ModelsSoumya Sanyal, Xiang RenEMNLP 2021 · 34 citations
Related papers
- Improving LLM Attributions with Randomized Path-IntegrationOren Barkan, Yehonatan Elisha, Yonatan Toib, Jonathan Weill et al.EMNLP Findings 2024 · 2 citations
- InteDisUX: Intepretation-Guided Discriminative User-Centric Explanation for Time SeriesViet-Hung Tran, Zichi Zhang, Tuan Dung Pham, Ngoc Phu Doan et al.AAAI 2025
- Integrated Decision Gradients: Compute Your Attributions Where the Model Makes Its DecisionChase Walker, Sumit Kumar Jha, Kenny Chen, Rickard EwetzAAAI 2024 · 25 citations
- Denoising Diffusion Path: Attribution Noise Reduction with An Auxiliary Diffusion ModelYiming Lei, Zilong Li, Junping Zhang, Hongming ShanNeurIPS 2024 · 9 citations
- Self-Guided Integrated Gradient Method for AttributionSabrina Henry, Alice Ruget, Stirling Scholes, Jonathan LeachCVPR Findings 2026
