ACL Findings2024
SMR: State Memory Replay for Long Sequence Modeling
Biqing Qi, Junqi Gao, Kaiyan Zhang, Dong Li, Jianxing Liu, Ligang Wu, Bowen Zhou
Abstract
Despite the promising performance of state space models (SSMs) in long sequence modeling, limitations still exist. Advanced SSMs like S5 and S6 (Mamba) in addressing non-uniform sampling, their recursive structures impede efficient SSM computation via convolution. To overcome compatibility limitations in parallel convolutional computation, this paper proposes a novel non-recursive non-uniform sample processing strategy. Theoretical analysis of SSMs through the lens of Event-Triggered Control (ETC) theory reveals the Non-Stable State (NSS) problem, where deviations from sampling point requirements lead to error transmission and accumulation, causing the divergence of the SSM's hidden state. Our analysis further reveals that adjustments of input sequences with early memories can mitigate the NSS problem, achieving Sampling Step Adaptation (SSA). Building on this insight, we introduce a simple yet effective plug-and-play mechanism, State Memory Replay (SMR), which utilizes learnable memories to adjust the current state with multi-step information for generalization at sampling points different from those in the training data. This enables SSMs to stably model varying sampling points. Experiments on long-range modeling tasks in autoregressive language modeling and Long Range Arena demonstrate the general effectiveness of the SMR mechanism for a series of SSM models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Diagonal State Spaces are as Effective as Structured State Spaces cites Long Range Arena : A Benchmark for Efficient Transformers
- Diagonal State Spaces are as Effective as Structured State Spaces cites Efficiently Modeling Long Sequences with Structured State Spaces
- Diagonal State Spaces are as Effective as Structured State Spaces cites Language Models are Few-Shot Learners
- Efficiently Modeling Long Sequences with Structured State Spaces cites Long Range Arena : A Benchmark for Efficient Transformers
- Exploring Adversarial Robustness of Deep State Space Models cites Diagonal State Spaces are as Effective as Structured State Spaces
- Exploring Adversarial Robustness of Deep State Space Models cites Efficiently Modeling Long Sequences with Structured State Spaces
- Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization cites Efficiently Modeling Long Sequences with Structured State Spaces
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
- CodeSSM: Towards State Space Models for Code Understanding cites Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
Cited by top-tier papers3
- Exploring Adversarial Robustness of Deep State Space ModelsBiqing Qi, Yiang Luo, Junqi Gao, Pengfei Li et al.NeurIPS 2024 · 7 citations
- Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length GeneralizationErmo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang et al.ICML 2025
- CodeSSM: Towards State Space Models for Code UnderstandingShweta Verma, Abhinav Anand, Mira MeziniEMNLP 2025
Builds on13
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
- Long Range Arena : A Benchmark for Efficient TransformersYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen et al.ICLR 2021 · 881 citations
- Diagonal State Spaces are as Effective as Structured State SpacesAnkit Gupta, Albert Gu, Jonathan BerantNeurIPS 2022 · 546 citations
Related papers
- Block-Biased Mamba for Long-Range Sequence ProcessingAnnan Yu, N. Benjamin ErichsonNeurIPS 2025 · 10 citations
- Longhorn: State Space Models are Amortized Online LearnersBo Liu, Rui Wang, Lemeng Wu, Yihao Feng et al.ICLR 2025
- Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic dataTianyi Chen, Pengxiao Lin, Zhiwei Wang, Zhi-Qin John XuNeurIPS 2025 · 4 citations
- The Expressive Capacity of State Space Models: A Formal Language PerspectiveYash Raj Sarrof, Yana Veitsman, Michael HahnNeurIPS 2024 · 53 citations
- Bridging Expressivity and Scalability with Adaptive Unitary SSMsArjun Karuvally, Franz Nowak, T. Anderson Keller, Carmen Amo Alonso et al.NeurIPS 2025 · 8 citations
