Lune

EMNLP Findings2023

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

Shahar Katz, Yonatan Belinkov

Abstract

Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary, a transformation that makes them more human interpretable. In this paper, we investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mechanism. Based on our discoveries, we create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph, with nodes representing neurons or hidden states and edges representing the interactions between them. Our visualization simplifies huge amounts of data into easy-toread plots that can reflect the models' internal processing, uncovering the contribution of each component to the models' final prediction. Our visualization also unveils new insights about the role of layer norms as semantic filters that influence the models' output, and about neurons that are always activated during forward passes and act as regularization vectors. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

14 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Finding Transformer Circuits With Edge Pruning cites Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
  • Locating and Editing Factual Associations in GPT cites Mass-Editing Memory in a Transformer
  • Mass-Editing Memory in a Transformer cites Locating and Editing Factual Associations in GPT
  • Mass-Editing Memory in a Transformer cites Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  • Optimal ablation for interpretability cites Finding Transformer Circuits With Edge Pruning
  • Optimal ablation for interpretability cites Locating and Editing Factual Associations in GPT
  • Optimal ablation for interpretability cites Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  • Optimal ablation for interpretability cites Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
  • InversionView: A General-Purpose Method for Reading Information from Neural Activations cites Locating and Editing Factual Associations in GPT
  • InversionView: A General-Purpose Method for Reading Information from Neural Activations cites Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  • InversionView: A General-Purpose Method for Reading Information from Neural Activations cites Knowledge Circuits in Pretrained Transformers
  • InversionView: A General-Purpose Method for Reading Information from Neural Activations cites Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
  • Confidence Regulation Neurons in Language Models cites Locating and Editing Factual Associations in GPT
  • Confidence Regulation Neurons in Language Models cites Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  • Confidence Regulation Neurons in Language Models cites Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
  • Knowledge Circuits in Pretrained Transformers cites Finding Transformer Circuits With Edge Pruning
  • Knowledge Circuits in Pretrained Transformers cites Locating and Editing Factual Associations in GPT
  • Knowledge Circuits in Pretrained Transformers cites Mass-Editing Memory in a Transformer
  • Knowledge Circuits in Pretrained Transformers cites Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space
  • Knowledge Circuits in Pretrained Transformers cites Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
  • Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small cites Locating and Editing Factual Associations in GPT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers14

  1. Finding Transformer Circuits With Edge PruningAdithya Bhaskar, Alexander Wettig, Dan Friedman, Danqi ChenNeurIPS 2024 · 72 citations
  2. Knowledge Circuits in Pretrained TransformersYunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang et al.NeurIPS 2024 · 71 citations
  3. Confidence Regulation Neurons in Language ModelsAlessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov et al.NeurIPS 2024 · 68 citations
  4. Optimal ablation for interpretabilityMaximilian Li, Lucas JansonNeurIPS 2024 · 32 citations
  5. InversionView: A General-Purpose Method for Reading Information from Neural ActivationsXinting Huang, Madhur Panwar, Navin Goyal, Michael HahnNeurIPS 2024 · 10 citations

Builds on9

  1. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  2. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary SpaceMor Geva, Avi Caciularu, Kevin Ro Wang, Yoav GoldbergEMNLP 2022 · 92 citations
  3. All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational QualityWilliam Timkey, Marten van SchijndelEMNLP 2021 · 59 citations
  4. Mass-Editing Memory in a TransformerKevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov et al.ICLR 2023 · 52 citations
  5. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 SmallKevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris et al.ICLR 2023 · 50 citations

Related papers

  1. Backward Lens: Projecting Language Model Gradients into the Vocabulary SpaceShahar Katz, Yonatan Belinkov, Mor Geva, Lior WolfEMNLP 2024 · 2 citations
  2. FlowPrune: Accelerating Attention Flow Calculation by Pruning Flow NetworkShuo Xu, Yu Chen, Shuxia Lin, Xin Geng et al.NeurIPS 2025 · 1 citation
  3. VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language TransformersEstelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu et al.CVPR 2022 · 34 citations
  4. Head Pursuit: Probing Attention Specialization in Multimodal TransformersLorenzo Basile, Valentino Maiorca, Diego Doimo, Francesco Locatello et al.NeurIPS 2025 · 21 citations
  5. Incorporating Residual and Normalization Layers into Analysis of Masked Language ModelsGoro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro InuiEMNLP 2021 · 28 citations
Dusk over the sea between two cliffs drawn in fine vertical lines