Lune

ACL Findings2025

Revealing and Mitigating the Local Pattern Shortcuts of Mamba

Wangjie You, Zecheng Tang, Juntao Li, Lili Yao, Min Zhang

Abstract

Large language models (LLMs) have advanced significantly due to the attention mechanism, but their quadratic complexity and linear memory demands limit their performance on longcontext tasks. Recently, researchers introduced Mamba, an advanced model built upon State Space Models (SSMs) that offers linear complexity and constant memory. Although Mamba is reported to match or surpass the performance of attention-based models, our analysis reveals a performance gap: Mamba excels in tasks that involve localized key information but faces challenges with tasks that require handling distributed key information. Our controlled experiments suggest that the inconsistency arises from Mamba's reliance on local pattern shortcuts across model scales (10M to 1.4B), which enable Mamba to remember local key information within its limited memory but hinder its ability to retain more dispersed information. Therefore, we introduce a global gate module into the Mamba model to address this issue. Experiments on extensive synthetic tasks, as well as real-world tasks, demonstrate the effectiveness of our method. Notably, with the introduction of only 4M extra parameters, our approach enables the Mamba model (130M) to achieve a significant improvement on tasks with distributed information, increasing its performance from below 5% to 80%.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers cites HiPPO: Recurrent Memory with Optimal Polynomial Projections
  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality cites HiPPO: Recurrent Memory with Optimal Polynomial Projections
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. LOGO - Long cOntext aliGnment via efficient preference OptimizationZecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu et al.ICML 2025

Builds on20

  1. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  2. Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
  3. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space DualityTri Dao, Albert GuICML 2024 · 1,407 citations
  4. Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space LayersAlbert Gu, Isys Johnson, Karan Goel, Khaled Saab et al.NeurIPS 2021 · 1,280 citations
  5. HiPPO: Recurrent Memory with Optimal Polynomial ProjectionsAlbert Gu, Tri Dao, Stefano Ermon, Atri Rudra et al.NeurIPS 2020 · 1,100 citations

Related papers

  1. LongMamba: Enhancing Mamba's Long-Context Capabilities via Training-Free Receptive Field EnlargementZhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong et al.ICLR 2025
  2. Mamba Modulation: On the Length Generalization of Mamba ModelsPeng Lu, Jerry Huang, Qiuhao Zeng, Xinyu Wang et al.NeurIPS 2025 · 2 citations
  3. Routing Mamba: Scaling State Space Models with Mixture-of-Experts ProjectionZheng Zhan, Liliang Ren, Shuohang Wang, Liyuan Liu et al.NeurIPS 2025 · 4 citations
  4. TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language ModelYixing Li, Ruobing Xie, Zhen Yang, Xingwu Sun et al.AAAI 2026 · 3 citations
  5. Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic dataTianyi Chen, Pengxiao Lin, Zhiwei Wang, Zhi-Qin John XuNeurIPS 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines