  Transformer, first applied to the field of natural language processing, is a
type of deep neural network mainly based on the self-attention mechanism.
Thanks to its strong representation capabilities, researchers are looking at
ways to apply transformer to computer vision tasks. In a variety of visual
benchmarks, transformer-based models perform similar to or better than other
types of networks such as convolutional and recurrent neural networks. Given
its high performance and less need for vision-specific inductive bias,
transformer is receiving more and more attention from the computer vision
community. In this paper, we review these vision transformer models by
categorizing them in different tasks and analyzing their advantages and
disadvantages. The main categories we explore include the backbone network,
high/mid-level vision, low-level vision, and video processing. We also include
efficient transformer methods for pushing transformer into real device-based
applications. Furthermore, we also take a brief look at the self-attention
mechanism in computer vision, as it is the base component in transformer.
Toward the end of this paper, we discuss the challenges and provide several
further research directions for vision transformers.
