  Self-attention has become a defacto choice for capturing global context in
various vision applications. However, its quadratic computational complexity
with respect to image resolution limits its use in real-time applications,
especially for deployment on resource-constrained mobile devices. Although
hybrid approaches have been proposed to combine the advantages of convolutions
and self-attention for a better speed-accuracy trade-off, the expensive matrix
multiplication operations in self-attention remain a bottleneck. In this work,
we introduce a novel efficient additive attention mechanism that effectively
replaces the quadratic matrix multiplication operations with linear
element-wise multiplications. Our design shows that the key-value interaction
can be replaced with a linear layer without sacrificing any accuracy. Unlike
previous state-of-the-art methods, our efficient formulation of self-attention
enables its usage at all stages of the network. Using our proposed efficient
additive attention, we build a series of models called "SwiftFormer" which
achieves state-of-the-art performance in terms of both accuracy and mobile
inference speed. Our small variant achieves 78.5% top-1 ImageNet-1K accuracy
with only 0.8 ms latency on iPhone 14, which is more accurate and 2x faster
compared to MobileViT-v2. Code: https://github.com/Amshaker/SwiftFormer
