Forgetting Transformer: Softmax Attention with a Forget Gate
Zhixuan Lin, Evgenii Nikishin, Xu Owen He, Aaron C. Courville
Abstract
An essential component of modern recurrent sequence models is the forget gate. While Transformers do not have an explicit recurrent form, we show that a forget gate can be naturally incorporated into Transformers by down-weighting the unnormalized attention scores in a data-dependent way. We name this attention mechanism Forgetting Attention and the resulting model the Forgetting Transformer (FoX). We show that FoX outperforms the Transformer on longcontext language modeling, length extrapolation, and short-context downstream tasks, while performing on par with the Transformer on long-context downstream tasks. Moreover, it is compatible with the FlashAttention algorithm and does not require any positional embeddings. Several analyses, including the needle-in-the-haystack test, show that FoX also retains the Transformer's superior long-context capabilities over recurrent sequence models such as Mamba-2, HGRN2, and DeltaNet. We also introduce a "Pro" block design that incorporates some common architectural components in recurrent sequence models and find it significantly improves the performance of both FoX and the Transformer. Our code is available at https://github.com/zhixuan-lin/ forgetting-transformer.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
20 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products cites PIQA: Reasoning about Physical Commonsense in Natural Language
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products cites Efficiently Modeling Long Sequences with Structured State Spaces
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Test-Time Training Done Right cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Test-Time Training Done Right cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Log-Linear Attention cites DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
- Log-Linear Attention cites Efficiently Modeling Long Sequences with Structured State Spaces
- Log-Linear Attention cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Log-Linear Attention cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- MesaNet: Sequence Modeling by Locally Optimal Test-Time Training cites PIQA: Reasoning about Physical Commonsense in Natural Language
- MesaNet: Sequence Modeling by Locally Optimal Test-Time Training cites Efficiently Modeling Long Sequences with Structured State Spaces
- MesaNet: Sequence Modeling by Locally Optimal Test-Time Training cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- MesaNet: Sequence Modeling by Locally Optimal Test-Time Training cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites Efficiently Modeling Long Sequences with Structured State Spaces
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Cited by top-tier papers20
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-FreeZihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang et al.NeurIPS 2025 · 336 citations
- Test-Time Training Done RightTianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang et al.ICLR 2026 · 127 citations
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder ProductsJulien Siems, Timur Carstensen, Arber Zela, Frank Hutter et al.NeurIPS 2025 · 75 citations
- MesaNet: Sequence Modeling by Locally Optimal Test-Time TrainingJohannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari et al.ICLR 2026 · 45 citations
- Log-Linear AttentionHan Guo, Songlin Yang, Tarushii Goel, Eric P. Xing et al.ICLR 2026 · 41 citations
Builds on29
- Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityTri Dao, Albert GuICML 2024 · 1,407 citations
- Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 citations
Related papers
- Transformer Quality in Linear TimeWeizhe Hua, Zihang Dai, Hanxiao Liu, Quoc V. LeICML 2022 · 335 citations
- FiX: Introducing Fine-grained Forget Gate into Softmax AttentionRunzhong Li, Renjie Liu, Qing Li, Bo TangICML 2026
- Gated Linear Attention Transformers with Hardware-Efficient TrainingSonglin Yang, Bailin Wang, Yikang Shen, Rameswar Panda et al.ICML 2024 · 390 citations
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence ModelingXiuying Wei, Anunay Yadav, Razvan Pascanu, Caglar GulcehreNeurIPS 2025 · 3 citations
- Gated Delta Networks: Improving Mamba2 with Delta RuleSonglin Yang, Jan Kautz, Ali HatamizadehICLR 2025
