  Self-attention in Transformers comes with a high computational cost because
of their quadratic computational complexity, but their effectiveness in
addressing problems in language and vision has sparked extensive research aimed
at enhancing their efficiency. However, diverse experimental conditions,
spanning multiple input domains, prevent a fair comparison based solely on
reported results, posing challenges for model selection. To address this gap in
comparability, we perform a large-scale benchmark of more than 45 models for
image classification, evaluating key efficiency aspects, including accuracy,
speed, and memory usage. Our benchmark provides a standardized baseline for
efficiency-oriented transformers. We analyze the results based on the Pareto
front -- the boundary of optimal models. Surprisingly, despite claims of other
models being more efficient, ViT remains Pareto optimal across multiple
metrics. We observe that hybrid attention-CNN models exhibit remarkable
inference memory- and parameter-efficiency. Moreover, our benchmark shows that
using a larger model in general is more efficient than using higher resolution
images. Thanks to our holistic evaluation, we provide a centralized resource
for practitioners and researchers, facilitating informed decisions when
selecting or developing efficient transformers.
