  Self-attention based models such as vision transformers (ViTs) have emerged
as a very competitive architecture alternative to convolutional neural networks
(CNNs) in computer vision. Despite increasingly stronger variants with
ever-higher recognition accuracies, due to the quadratic complexity of
self-attention, existing ViTs are typically demanding in computation and model
size. Although several successful design choices (e.g., the convolutions and
hierarchical multi-stage structure) of prior CNNs have been reintroduced into
recent ViTs, they are still not sufficient to meet the limited resource
requirements of mobile devices. This motivates a very recent attempt to develop
light ViTs based on the state-of-the-art MobileNet-v2, but still leaves a
performance gap behind. In this work, pushing further along this under-studied
direction we introduce EdgeViTs, a new family of light-weight ViTs that, for
the first time, enable attention-based vision models to compete with the best
light-weight CNNs in the tradeoff between accuracy and on-device efficiency.
This is realized by introducing a highly cost-effective local-global-local
(LGL) information exchange bottleneck based on optimal integration of
self-attention and convolutions. For device-dedicated evaluation, rather than
relying on inaccurate proxies like the number of FLOPs or parameters, we adopt
a practical approach of focusing directly on on-device latency and, for the
first time, energy efficiency. Specifically, we show that our models are
Pareto-optimal when both accuracy-latency and accuracy-energy trade-offs are
considered, achieving strict dominance over other ViTs in almost all cases and
competing with the most efficient CNNs. Code is available at
https://github.com/saic-fi/edgevit.
