  Vision transformers (ViTs) have demonstrated great potential in various
visual tasks, but suffer from expensive computational and memory cost problems
when deployed on resource-constrained devices. In this paper, we introduce a
ternary vision transformer (TerViT) to ternarize the weights in ViTs, which are
challenged by the large loss surface gap between real-valued and ternary
parameters. To address the issue, we introduce a progressive training scheme by
first training 8-bit transformers and then TerViT, and achieve a better
optimization than conventional methods. Furthermore, we introduce channel-wise
ternarization, by partitioning each matrix to different channels, each of which
is with an unique distribution and ternarization interval. We apply our methods
to popular DeiT and Swin backbones, and extensive results show that we can
achieve competitive performance. For example, TerViT can quantize Swin-S to
13.1MB model size while achieving above 79% Top-1 accuracy on ImageNet dataset.
