Local Interpretation of Transformer Based on Linear Decomposition
Sen Yang, Shujian Huang, Wei Zou, Jianbing Zhang, Xinyu Dai, Jiajun Chen
Abstract
In recent years, deep neural networks (DNNs) have achieved state-of-the-art performance on a wide range of tasks. However, limitations in interpretability have hindered their applications in the real world. This work proposes to interpret neural networks by linear decomposition and finds that the ReLU-activated Transformer can be considered as a linear model on a single input. We further leverage the linearity of the model and propose a linear decomposition of the model output to generate local explanations. Our evaluation of sentiment classification and machine translation shows that our method achieves competitive performance in efficiency and fidelity of explanation. In addition, we demonstrate the potential of our approach in applications with examples of error analysis on multiple tasks. 1 1 We release our algorithm toolkit at https://github. com/DoubleVII/pydec .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety cites Incorporating Residual and Normalization Layers into Analysis of Masked Language Models
- Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection cites Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer cites Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer cites Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
Cited by top-tier papers2
- DePass: Unified Feature Attributing by Simple Decomposed Forward PassXiangyu Hong, Che Jiang, Kai Tian, Biqing Qi et al.NeurIPS 2025 · 4 citations
- Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM SafetySeongmin Lee, Aeree Cho, Grace C. Kim, Shengyun Peng et al.EMNLP 2025 · 1 citation
Builds on7
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Self-Attention Attribution: Interpreting Information Interactions Inside TransformerYaru Hao, Li Dong, Furu Wei, Ke XuAAAI 2021 · 282 citations
- Generating Hierarchical Explanations on Text Classification via Feature Interaction DetectionHanjie Chen, Guangtao Zheng, Yangfeng JiACL 2020 · 85 citations
- Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence ModelsXisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue et al.ICLR 2020 · 55 citations
- Incorporating Residual and Normalization Layers into Analysis of Masked Language ModelsGoro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro InuiEMNLP 2021 · 28 citations
Related papers
- Leveraging Sparse Linear Layers for Debuggable Deep NetworksEric Wong, Shibani Santurkar, Aleksander MadryICML 2021 · 101 citations
- TropEx: An Algorithm for Extracting Linear Terms in Deep Neural NetworksMartin Trimmel, Henning Petzka, Cristian SminchisescuICLR 2021 · 15 citations
- Efficient Automated Circuit Discovery in Transformers using Contextual DecompositionAliyah R. Hsu, Georgia Zhou, Yeshwanth Cherapanamjeri, Yaxuan Huang et al.ICLR 2025
- XAI for Transformers: Better Explanations through Conservative PropagationAmeen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon et al.ICML 2022 · 144 citations
- Corrupting Neuron Explanations of Deep Visual FeaturesDivyansh Srivastava, Tuomas P. Oikarinen, Tsui-Wei WengICCV 2023 · 3 citations
