  We attempt to reduce the computational costs in vision transformers (ViTs),
which increase quadratically in the token number. We present a novel training
paradigm that trains only one ViT model at a time, but is capable of providing
improved image recognition performance with various computational costs. Here,
the trained ViT model, termed super vision transformer (SuperViT), is empowered
with the versatile ability to solve incoming patches of multiple sizes as well
as preserve informative tokens with multiple keeping rates (the ratio of
keeping tokens) to achieve good hardware efficiency for inference, given that
the available hardware resources often change from time to time. Experimental
results on ImageNet demonstrate that our SuperViT can considerably reduce the
computational costs of ViT models with even performance increase. For example,
we reduce 2x FLOPs of DeiT-S while increasing the Top-1 accuracy by 0.2% and
0.7% for 1.5x reduction. Also, our SuperViT significantly outperforms existing
studies on efficient vision transformers. For example, when consuming the same
amount of FLOPs, our SuperViT surpasses the recent state-of-the-art (SOTA) EViT
by 1.1% when using DeiT-S as their backbones. The project of this work is made
publicly available at https://github.com/lmbxmu/SuperViT.
