  The transformer architectures with attention mechanisms have obtained success
in Nature Language Processing (NLP), and Vision Transformers (ViTs) have
recently extended the application domains to various vision tasks. While
achieving high performance, ViTs suffer from large model size and high
computation complexity that hinders the deployment of them on edge devices. To
achieve high throughput on hardware and preserve the model accuracy
simultaneously, we propose VAQF, a framework that builds inference accelerators
on FPGA platforms for quantized ViTs with binary weights and low-precision
activations. Given the model structure and the desired frame rate, VAQF will
automatically output the required quantization precision for activations as
well as the optimized parameter settings of the accelerator that fulfill the
hardware requirements. The implementations are developed with Vivado High-Level
Synthesis (HLS) on the Xilinx ZCU102 FPGA board, and the evaluation results
with the DeiT-base model indicate that a frame rate requirement of 24 frames
per second (FPS) is satisfied with 8-bit activation quantization, and a target
of 30 FPS is met with 6-bit activation quantization. To the best of our
knowledge, this is the first time quantization has been incorporated into ViT
acceleration on FPGAs with the help of a fully automatic framework to guide the
quantization strategy on the software side and the accelerator implementations
on the hardware side given the target frame rate. Very small compilation time
cost is incurred compared with quantization training, and the generated
accelerators show the capability of achieving real-time execution for
state-of-the-art ViT models on FPGAs.
