  While vision transformers (ViTs) have continuously achieved new milestones in
the field of computer vision, their sophisticated network architectures with
high computation and memory costs have impeded their deployment on
resource-limited edge devices. In this paper, we propose a hardware-efficient
image-adaptive token pruning framework called HeatViT for efficient yet
accurate ViT acceleration on embedded FPGAs. By analyzing the inherent
computational patterns in ViTs, we first design an effective attention-based
multi-head token selector, which can be progressively inserted before
transformer blocks to dynamically identify and consolidate the non-informative
tokens from input images. Moreover, we implement the token selector on hardware
by adding miniature control logic to heavily reuse existing hardware components
built for the backbone ViT. To improve the hardware efficiency, we further
employ 8-bit fixed-point quantization, and propose polynomial approximations
with regularization effect on quantization error for the frequently used
nonlinear functions in ViTs. Finally, we propose a latency-aware multi-stage
training strategy to determine the transformer blocks for inserting token
selectors and optimize the desired (average) pruning rates for inserted token
selectors, in order to improve both the model accuracy and inference latency on
hardware. Compared to existing ViT pruning studies, under the similar
computation cost, HeatViT can achieve 0.7%$\sim$8.9% higher accuracy; while
under the similar model accuracy, HeatViT can achieve more than
28.4%$\sim$65.3% computation reduction, for various widely used ViTs, including
DeiT-T, DeiT-S, DeiT-B, LV-ViT-S, and LV-ViT-M, on the ImageNet dataset.
Compared to the baseline hardware accelerator, our implementations of HeatViT
on the Xilinx ZCU102 FPGA achieve 3.46$\times$$\sim$4.89$\times$ speedup.
