  Vision transformers (ViTs) have achieved remarkable performance in various
computer vision tasks. However, intensive memory and computation requirements
impede ViTs from running on resource-constrained edge devices. Due to the
non-normally distributed values after Softmax and GeLU, post-training
quantization on ViTs results in severe accuracy degradation. Moreover,
conventional methods fail to address the high channel-wise variance in
LayerNorm. To reduce the quantization loss and improve classification accuracy,
we propose a two-scaled post-training quantization scheme for vision
transformer (TSPTQ-ViT). We design the value-aware two-scaled scaling factors
(V-2SF) specialized for post-Softmax and post-GeLU values, which leverage the
bit sparsity in non-normal distribution to save bit-widths. In addition, the
outlier-aware two-scaled scaling factors (O-2SF) are introduced to LayerNorm,
alleviating the dominant impacts from outlier values. Our experimental results
show that the proposed methods reach near-lossless accuracy drops (<0.5%) on
the ImageNet classification task under 8-bit fully quantized ViTs.
