  We introduce A-ViT, a method that adaptively adjusts the inference cost of
vision transformer (ViT) for images of different complexity. A-ViT achieves
this by automatically reducing the number of tokens in vision transformers that
are processed in the network as inference proceeds. We reformulate Adaptive
Computation Time (ACT) for this task, extending halting to discard redundant
spatial tokens. The appealing architectural properties of vision transformers
enables our adaptive token reduction mechanism to speed up inference without
modifying the network architecture or inference hardware. We demonstrate that
A-ViT requires no extra parameters or sub-network for halting, as we base the
learning of adaptive halting on the original network parameters. We further
introduce distributional prior regularization that stabilizes training compared
to prior ACT approaches. On the image classification task (ImageNet1K), we show
that our proposed A-ViT yields high efficacy in filtering informative spatial
features and cutting down on the overall compute. The proposed method improves
the throughput of DeiT-Tiny by 62% and DeiT-Small by 38% with only 0.3%
accuracy drop, outperforming prior art by a large margin. Project page at
https://a-vit.github.io/
