  Mamba is an effective state space model with linear computation complexity.
It has recently shown impressive efficiency in dealing with high-resolution
inputs across various vision tasks. In this paper, we reveal that the powerful
Mamba model shares surprising similarities with linear attention Transformer,
which typically underperform conventional Transformer in practice. By exploring
the similarities and disparities between the effective Mamba and subpar linear
attention Transformer, we provide comprehensive analyses to demystify the key
factors behind Mamba's success. Specifically, we reformulate the selective
state space model and linear attention within a unified formulation, rephrasing
Mamba as a variant of linear attention Transformer with six major distinctions:
input gate, forget gate, shortcut, no attention normalization, single-head, and
modified block design. For each design, we meticulously analyze its pros and
cons, and empirically evaluate its impact on model performance in vision tasks.
Interestingly, the results highlight the forget gate and block design as the
core contributors to Mamba's success, while the other four designs are less
crucial. Based on these findings, we propose a Mamba-Like Linear Attention
(MLLA) model by incorporating the merits of these two key designs into linear
attention. The resulting model outperforms various vision Mamba models in both
image classification and high-resolution dense prediction tasks, while enjoying
parallelizable computation and fast inference speed. Code is available at
https://github.com/LeapLabTHU/MLLA.
