  Vision Transformers (ViTs) take all the image patches as tokens and construct
multi-head self-attention (MHSA) among them. Complete leverage of these image
tokens brings redundant computations since not all the tokens are attentive in
MHSA. Examples include that tokens containing semantically meaningless or
distractive image backgrounds do not positively contribute to the ViT
predictions. In this work, we propose to reorganize image tokens during the
feed-forward process of ViT models, which is integrated into ViT during
training. For each forward inference, we identify the attentive image tokens
between MHSA and FFN (i.e., feed-forward network) modules, which is guided by
the corresponding class token attention. Then, we reorganize image tokens by
preserving attentive image tokens and fusing inattentive ones to expedite
subsequent MHSA and FFN computations. To this end, our method EViT improves
ViTs from two perspectives. First, under the same amount of input image tokens,
our method reduces MHSA and FFN computation for efficient inference. For
instance, the inference speed of DeiT-S is increased by 50% while its
recognition accuracy is decreased by only 0.3% for ImageNet classification.
Second, by maintaining the same computational cost, our method empowers ViTs to
take more image tokens as input for recognition accuracy improvement, where the
image tokens are from higher resolution images. An example is that we improve
the recognition accuracy of DeiT-S by 1% for ImageNet classification at the
same computational cost of a vanilla DeiT-S. Meanwhile, our method does not
introduce more parameters to ViTs. Experiments on the standard benchmarks show
the effectiveness of our method. The code is available at
https://github.com/youweiliang/evit
