  Vision Transformers (ViT) have made many breakthroughs in computer vision
tasks. However, considerable redundancy arises in the spatial dimension of an
input image, leading to massive computational costs. Therefore, We propose a
coarse-to-fine vision transformer (CF-ViT) to relieve computational burden
while retaining performance in this paper. Our proposed CF-ViT is motivated by
two important observations in modern ViT models: (1) The coarse-grained patch
splitting can locate informative regions of an input image. (2) Most images can
be well recognized by a ViT model in a small-length token sequence. Therefore,
our CF-ViT implements network inference in a two-stage manner. At coarse
inference stage, an input image is split into a small-length patch sequence for
a computationally economical classification. If not well recognized, the
informative patches are identified and further re-split in a fine-grained
granularity. Extensive experiments demonstrate the efficacy of our CF-ViT. For
example, without any compromise on performance, CF-ViT reduces 53% FLOPs of
LV-ViT, and also achieves 2.01x throughput.
