ACL Findings2023
On the Expressivity Role of LayerNorm in Transformers' Attention
Shaked Brody, Uri Alon, Eran Yahav
Abstract
Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. In this paper, we show that LayerNorm is crucial to the expressivity of the multi-head attention layer that follows it. This is in contrast to the common belief that LayerNorm's only role is to normalize the activations during the forward pass, and their gradients during the backward pass.
We consider a geometric interpretation of LayerNorm and show that it consists of two components: (a) projection of the input vectors to a d -1 space that is orthogonal to the [1, 1, ..., 1] vector, and (b) scaling of all vectors to the same norm of √ d. We show that each of these components is important for the attention layer that follows it in Transformers: (a) projection allows the attention mechanism to create an attention query that attends to all keys equally, offloading the need to learn this operation by the attention; and (b) scaling allows each key to potentially receive the highest attention, and prevents keys from being "un-select-able". We show empirically that Transformers do indeed benefit from these properties of LayeNorm in general language modeling and even in computing simple functions such as "majority". Our code is available at https://github.com/tech-srl/ layer_norm_expressivity_role.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Impact of Layer Norm on Memorization and Generalization in Transformers cites On the Role of Attention Masks and LayerNorm in Transformers
- Impact of Layer Norm on Memorization and Generalization in Transformers cites On Layer Normalization in the Transformer Architecture
- On the Role of Attention Masks and LayerNorm in Transformers cites On Layer Normalization in the Transformer Architecture
Cited by top-tier papers10
- On the Role of Attention Masks and LayerNorm in TransformersXinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka et al.NeurIPS 2024 · 54 citations
- Prompting a Pretrained Transformer Can Be a Universal ApproximatorAleksandar Petrov, Philip Torr, Adel BibiICML 2024 · 19 citations
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression PerspectiveQishuai Wen, Chun-Guang LiNeurIPS 2024 · 8 citations
- Impact of Layer Norm on Memorization and Generalization in TransformersRishi Singhal, Jung-Eun KimNeurIPS 2025 · 4 citations
- Stealth edits to large language modelsOliver J. Sutton, Qinghua Zhou, Wei Wang, Desmond J. Higham et al.NeurIPS 2024 · 4 citations
Builds on2
Related papers
- Incorporating Residual and Normalization Layers into Analysis of Masked Language ModelsGoro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro InuiEMNLP 2021 · 28 citations
- XAI for Transformers: Better Explanations through Conservative PropagationAmeen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon et al.ICML 2022 · 144 citations
- VISIT: Visualizing and Interpreting the Semantic Information Flow of TransformersShahar Katz, Yonatan BelinkovEMNLP Findings 2023 · 12 citations
- Small Transformers Don’t Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and Implications for Mechanistic InterpretabilityLuca Baroni, Galvin Khara, Joachim Schaeffer, Marat Subkhankulov et al.ICLR 2026 · 8 citations
- Attention is Not Only a Weight: Analyzing Transformers with Vector NormsGoro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro InuiEMNLP 2020 · 138 citations
