  Post-training quantization (PTQ), which only requires a tiny dataset for
calibration without end-to-end retraining, is a light and practical model
compression technique. Recently, several PTQ schemes for vision transformers
(ViTs) have been presented; unfortunately, they typically suffer from
non-trivial accuracy degradation, especially in low-bit cases. In this paper,
we propose RepQ-ViT, a novel PTQ framework for ViTs based on quantization scale
reparameterization, to address the above issues. RepQ-ViT decouples the
quantization and inference processes, where the former employs complex
quantizers and the latter employs scale-reparameterized simplified quantizers.
This ensures both accurate quantization and efficient inference, which
distinguishes it from existing approaches that sacrifice quantization
performance to meet the target hardware. More specifically, we focus on two
components with extreme distributions: post-LayerNorm activations with severe
inter-channel variation and post-Softmax activations with power-law features,
and initially apply channel-wise quantization and log$\sqrt{2}$ quantization,
respectively. Then, we reparameterize the scales to hardware-friendly
layer-wise quantization and log2 quantization for inference, with only slight
accuracy or computational costs. Extensive experiments are conducted on
multiple vision tasks with different model variants, proving that RepQ-ViT,
without hyperparameters and expensive reconstruction procedures, can outperform
existing strong baselines and encouragingly improve the accuracy of 4-bit PTQ
of ViTs to a usable level. Code is available at
https://github.com/zkkli/RepQ-ViT.
