Lune

ICLR2025Top-tier venue

MambaExtend: A Training-Free Approach to Improve Long Context Extension of Mamba

Seyedarmin Azizi, Souvik Kundu, Mohammad Erfan Sadeghi, Massoud Pedram

Abstract

The inherent quadratic complexity of the attention mechanism in transformer models has driven the research community to explore alternative architectures with sub-quadratic complexity, such as state-space models. Mamba has established itself as a leading model within this emerging paradigm, achieving stateof-the-art results in various language modeling benchmarks. However, despite its impressive performance, Mamba's effectiveness is limited by its pre-training context length, resulting in a pronounced degradation when the model is tasked with handling longer contexts. Our investigation reveals that Mamba's inability to generalize effectively to long contexts is primarily due to the out-of-distribution (OOD) discretization steps. To address this critical limitation, we introduce Mam-baExtend, a novel framework designed to significantly enhance the context extension capabilities of Mamba. Specifically, MambaExtend leverages a training-free approach to calibrate only the scaling factors of discretization modules for different layers. We demonstrate both gradient-based and gradient-free zeroth-order optimization to learn the optimal scaling factors for each Mamba layer, requiring orders of magnitude fewer updates as opposed to the parameter fine-tuning-based alternatives. Using this approach, we achieve a training-free context extension of up to 32×, expanding the context from 2k to 64k tokens with minimal increases in perplexity. In contrast to existing fine-tuning methods, MambaExtend selectively calibrates the scaling factors, requiring up to ∼5.42 * 10 6 × fewer parameter updates and incurring up to 3.87× lower peak memory usage, while delivering comparable or superior long-context performance across multiple tasks. Codes and checkpoints are available here 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation cites Efficiently Modeling Long Sequences with Structured State Spaces
  • Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  • Understanding and Improving Length Generalization in Recurrent Models cites Efficiently Modeling Long Sequences with Structured State Spaces
  • Understanding and Improving Length Generalization in Recurrent Models cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  • Efficiently Modeling Long Sequences with Structured State Spaces cites HiPPO: Recurrent Memory with Optimal Polynomial Projections
  • HiPPO: Recurrent Memory with Optimal Polynomial Projections cites Reformer: The Efficient Transformer
  • HiPPO: Recurrent Memory with Optimal Polynomial Projections cites Compressive Transformers for Long-Range Sequence Modelling
  • From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation cites Efficiently Modeling Long Sequences with Structured State Spaces
  • From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation cites Reformer: The Efficient Transformer
  • From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation cites Compressive Transformers for Long-Range Sequence Modelling
  • From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  • Mamba Modulation: On the Length Generalization of Mamba Models cites Efficiently Modeling Long Sequences with Structured State Spaces
  • Mamba Modulation: On the Length Generalization of Mamba Models cites HiPPO: Recurrent Memory with Optimal Polynomial Projections
  • Mamba Modulation: On the Length Generalization of Mamba Models cites Reformer: The Efficient Transformer
  • Mamba Modulation: On the Length Generalization of Mamba Models cites Compressive Transformers for Long-Range Sequence Modelling
  • Mamba Modulation: On the Length Generalization of Mamba Models cites Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites Efficiently Modeling Long Sequences with Structured State Spaces
  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites HiPPO: Recurrent Memory with Optimal Polynomial Projections
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Ouromamba: a Data-Free Quantization Framework for Vision MambaAkshat Ramachandran, Mingyu Lee, Huan Xu, Souvik Kundu et al.ICCV 2025 · 3 citations
  2. Mamba Modulation: On the Length Generalization of Mamba ModelsPeng Lu, Jerry Huang, Qiuhao Zeng, Xinyu Wang et al.NeurIPS 2025 · 2 citations
  3. Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta ModulationYehjin Shin, Seojin Kim, Noseong ParkICLR 2026
  4. Understanding and Improving Length Generalization in Recurrent ModelsRicardo Buitrago Ruiz, Albert GuICML 2025
  5. From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position InterpolationHaochen Shen, Davis Wertheimer, Zheng Wang, Garrett Goon et al.ICLR 2026

Builds on16

  1. Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
  2. Reformer: The Efficient TransformerNikita Kitaev, Lukasz Kaiser, Anselm LevskayaICLR 2020 · 2,878 citations
  3. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityTri Dao, Albert GuICML 2024 · 1,407 citations
  4. HiPPO: Recurrent Memory with Optimal Polynomial ProjectionsAlbert Gu, Tri Dao, Stefano Ermon, Atri Rudra et al.NeurIPS 2020 · 1,100 citations
  5. Compressive Transformers for Long-Range Sequence ModellingJack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier et al.ICLR 2020 · 833 citations

Related papers

  1. LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field EnlargementZhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong et al.ICLR 2025
  2. DeciMamba: Exploring the Length Extrapolation Potential of MambaAssaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen et al.ICLR 2025
  3. Revealing and Mitigating the Local Pattern Shortcuts of MambaWangjie You, Zecheng Tang, Juntao Li, Lili Yao et al.ACL Findings 2025 · 1 citation
  4. Sparsifying MambaAn Wang, Ruobing Xie, Shuaipeng Li, Xingwu Sun et al.EMNLP Findings 2025 · 1 citation
  5. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language ModelingLiliang Ren, Yang Liu, Yadong Lu, Yelong Shen et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines