MambaOut: Do We Really Need Mamba for Vision?

In memory of Kobe Bryant "What can I say, Mamba out." -Kobe Bryant's NBA farewell speech in 2016. Linear Linear Conv σ SSM Linear σ Linear Linear Conv Linear σ Gated CNN block (e.g. Our MambaOut) Mamba block (e.g. Vision Mamba) (a) 0 5 10 15 20 25 MACs (G) 76 78 80 82 84 86 ImageNet Top-1 Accuracy (%) 20M 40M 80M Model Size MambaOut Vision Mamba VMamba PlainMamba Accuracy vs. MACs vs. Model Size (b) Figure 1: (a) Architecture of Gated CNN [18] and Mamba [25] blocks (omitting Normalization and shortcut). The Mamba block extends the Gated CNN with an additional state space model (SSM). As will be conceptually discussed in Section 3, SSM is not necessary for image classification on ImageNet [19, 66]. To empirically verify this claim, we stack Gated CNN blocks to build a series of models named MambaOut. (b) MambaOut outperforms visual Mamba models, e.g., Vision Mamhba [104], VMamba [50] and PlainMamba [88], on ImageNet image classification. Abstract -Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism and subsequently applied to vision tasks 1 . Nevertheless, the performance of Mamba for vision is often underwhelming when compared with convolutional and attention-based models. In this paper, we delve into the essence of Mamba, and conceptually conclude that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. For vision tasks, as image classification does not align with either characteristic, we hypothesize that Mamba is not necessary for this task; Detection and segmentation tasks are also not autoregressive, yet they adhere to the long-sequence characteristic, so we believe it is still worthwhile to explore Mamba's potential for these tasks. To empirically verify our hypotheses, we construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM. Experimental results strongly support our hypotheses. Specifically, our MambaOut model surpasses all visual Mamba models on ImageNet image classification, indicating that Mamba is indeed unnecessary for this task. As for detection and segmentation, MambaOut cannot match the performance of state-of-the-art visual Mamba models, demonstrating the potential of Mamba for long-sequence visual tasks. 1 The vision tasks we discuss in this paper include image classification on ImageNet [19, 66] , object detection & instance segmentation on COCO [48] and semantic segmentation on ADE20K [103] . Preprint. Under review.