  This paper studies the efficiency problem for visual transformers by
excavating redundant calculation in given networks. The recent transformer
architecture has demonstrated its effectiveness for achieving excellent
performance on a series of computer vision tasks. However, similar to that of
convolutional neural networks, the huge computational cost of vision
transformers is still a severe issue. Considering that the attention mechanism
aggregates different patches layer-by-layer, we present a novel patch slimming
approach that discards useless patches in a top-down paradigm. We first
identify the effective patches in the last layer and then use them to guide the
patch selection process of previous layers. For each layer, the impact of a
patch on the final output feature is approximated and patches with less impact
will be removed. Experimental results on benchmark datasets demonstrate that
the proposed method can significantly reduce the computational costs of vision
transformers without affecting their performances. For example, over 45% FLOPs
of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the
ImageNet dataset.
