Lune

ACL Findings2026

A Survey of Retentive Network

Haiqi Yang, Zhiyuan Li, Yi Chang, Yuan Wu

Abstract

The Retentive Network (RetNet) has recently emerged as a formidable successor to the Transformer architecture. Although the self-attention mechanism excels at capturing global dependencies, its inherent quadratic complexity imposes significant memory constraints and inhibits scalability during long-sequence modeling. To overcome these challenges, RetNet introduces an innovative retention mechanism that integrates the inductive bias of recurrent neural networks with the parallelizable training advantages of attention-based models. This unified representation allows RetNet to achieve constant-time inference and linear-time training without sacrificing representational capacity. Despite the growing body of research demonstrating the efficacy of RetNet across diverse fields such as natural language processing, computer vision, and time-series analysis, a systematic synthesis of the current literature is currently unavailable. This paper presents the first comprehensive survey of Retentive Networks through a detailed examination of its architectural foundations, core innovations, and specialized variants. Furthermore, we provide a multidisciplinary analysis of its applications ranging from basic sequence tasks to complex crossmodal scenarios. Finally, we offer prospective insights and suggest strategic avenues for future inquiry to facilitate the continued evolution of RetNet in both academic research and largescale industrial applications.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Hyena Hierarchy: Towards Larger Convolutional Language Models cites Efficiently Modeling Long Sequences with Structured State Spaces
  • Efficiently Modeling Long Sequences with Structured State Spaces cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • Parallelizing Linear Transformers with the Delta Rule over Sequence Length cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • Parallelizing Linear Transformers with the Delta Rule over Sequence Length cites Hyena Hierarchy: Towards Larger Convolutional Language Models
  • Parallelizing Linear Transformers with the Delta Rule over Sequence Length cites Efficiently Modeling Long Sequences with Structured State Spaces
  • Parallelizing Linear Transformers with the Delta Rule over Sequence Length cites Gated Linear Attention Transformers with Hardware-Efficient Training
  • Gated Linear Attention Transformers with Hardware-Efficient Training cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
  • Gated Linear Attention Transformers with Hardware-Efficient Training cites Hyena Hierarchy: Towards Larger Convolutional Language Models
  • Gated Linear Attention Transformers with Hardware-Efficient Training cites Efficiently Modeling Long Sequences with Structured State Spaces
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
  2. Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 citations
  3. Hyena Hierarchy: Towards Larger Convolutional Language ModelsMichael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu et al.ICML 2023 · 481 citations
  4. Parallelizing Linear Transformers with the Delta Rule over Sequence LengthSonglin Yang, Bailin Wang, Yu Zhang, Yikang Shen et al.NeurIPS 2024 · 412 citations
  5. Gated Linear Attention Transformers with Hardware-Efficient TrainingSonglin Yang, Bailin Wang, Yikang Shen, Rameswar Panda et al.ICML 2024 · 390 citations

Related papers

  1. Maskable Retentive Network for Video Moment RetrievalJingjing Hu, Dan Guo, Kun Li, Zhan Si et al.ACM MM 2024 · 7 citations
  2. RMT: Retentive Networks Meet Vision TransformersQihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu et al.CVPR 2024
  3. Transformer-Exclusive Cross-Modal Representation for Vision and LanguageAndrew Shin, Takuya NarihiraACL Findings 2021 · 2 citations
  4. Recurrent Memory TransformerAydar Bulatov, Yuri Kuratov, Mikhail BurtsevNeurIPS 2022 · 252 citations
  5. Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language ModelsS. M. Rafiuddin, Muntaha Nujat KhanEMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines