Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention
Arya Honarpisheh, Mustafa Bozdag, Octavia I. Camps, Mario Sznaier
Abstract
State-space models (SSMs) have recently emerged as a compelling alternative to Transformers for sequence modeling tasks. This paper presents a theoretical generalization analysis of selective SSMs, the core architectural component behind the Mamba model. We derive a novel covering number-based generalization bound for selective SSMs, building upon recent theoretical advances in the analysis of Transformer models. Using this result, we analyze how the spectral abscissa of the continuous-time state matrix influences the model's stability during training and its ability to generalize across sequence lengths. We empirically validate our findings on a synthetic majority task, the IMDb sentiment classification benchmark, and the ListOps task, demonstrating how our theoretical insights translate into practical model behavior.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 0a3899b8-ba61-4ea5-b3cf-c41b387b0fcfCited by top-tier papers2
- Block-Biased Mamba for Long-Range Sequence ProcessingAnnan Yu, N. Benjamin ErichsonNeurIPS 2025 · 10 citations
- A Theoretical Analysis of Mamba’s Training Dynamics: Filtering Relevant Features for Generalization in State Space ModelsMugunthan Shandirasegaran, Hongkang Li, Songyang Zhang, Meng Wang et al.ICLR 2026 · 3 citations
Builds on31
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
- Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityTri Dao, Albert GuICML 2024 · 1,407 citations
Related papers
- Mamba Modulation: On the Length Generalization of Mamba ModelsPeng Lu, Jerry Huang, Qiuhao Zeng, Xinyu Wang et al.NeurIPS 2025 · 2 citations
- Demystifying the Token Dynamics of Deep Selective State Space ModelsThieu N. Vo, Duy-Tung Pham, Xin T. Tong, Tan Minh NguyenICLR 2025
- On Feature Learning in Structured State Space ModelsLeena Chennuru Vankadara, Jin Xu, Moritz Haas, Volkan CevherNeurIPS 2024 · 10 citations
- Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall CapacityNingyuan Teresa Huang, Miguel Sarabia, Abhinav Moudgil, Pau Rodríguez et al.ICML 2025
- The Expressive Capacity of State Space Models: A Formal Language PerspectiveYash Raj Sarrof, Yana Veitsman, Michael HahnNeurIPS 2024 · 53 citations
