  Vision Transformers (ViTs) have achieved state-of-the-art performance on
various computer vision applications. However, these models have considerable
storage and computational overheads, making their deployment and efficient
inference on edge devices challenging. Quantization is a promising approach to
reducing model complexity, and the dyadic arithmetic pipeline can allow the
quantized models to perform efficient integer-only inference. Unfortunately,
dyadic arithmetic is based on the homogeneity condition in convolutional neural
networks, which is not applicable to the non-linear components in ViTs, making
integer-only inference of ViTs an open issue. In this paper, we propose I-ViT,
an integer-only quantization scheme for ViTs, to enable ViTs to perform the
entire computational graph of inference with integer arithmetic and
bit-shifting, and without any floating-point arithmetic. In I-ViT, linear
operations (e.g., MatMul and Dense) follow the integer-only pipeline with
dyadic arithmetic, and non-linear operations (e.g., Softmax, GELU, and
LayerNorm) are approximated by the proposed light-weight integer-only
arithmetic methods. More specifically, I-ViT applies the proposed Shiftmax and
ShiftGELU, which are designed to use integer bit-shifting to approximate the
corresponding floating-point operations. We evaluate I-ViT on various benchmark
models and the results show that integer-only INT8 quantization achieves
comparable (or even slightly higher) accuracy to the full-precision (FP)
baseline. Furthermore, we utilize TVM for practical hardware deployment on the
GPU's integer arithmetic units, achieving 3.72$\sim$4.11$\times$ inference
speedup compared to the FP model. Code of both Pytorch and TVM is released at
https://github.com/zkkli/I-ViT.
