  Transformers are considered one of the most important deep learning models
since 2018, in part because it establishes state-of-the-art (SOTA) records and
could potentially replace existing Deep Neural Networks (DNNs). Despite the
remarkable triumphs, the prolonged turnaround time of Transformer models is a
widely recognized roadblock. The variety of sequence lengths imposes additional
computing overhead where inputs need to be zero-padded to the maximum sentence
length in the batch to accommodate the parallel computing platforms. This paper
targets the field-programmable gate array (FPGA) and proposes a coherent
sequence length adaptive algorithm-hardware co-design for Transformer
acceleration. Particularly, we develop a hardware-friendly sparse attention
operator and a length-aware hardware resource scheduling algorithm. The
proposed sparse attention operator brings the complexity of attention-based
models down to linear complexity and alleviates the off-chip memory traffic.
The proposed length-aware resource hardware scheduling algorithm dynamically
allocates the hardware resources to fill up the pipeline slots and eliminates
bubbles for NLP tasks. Experiments show that our design has very small accuracy
loss and has 80.2 $\times$ and 2.6 $\times$ speedup compared to CPU and GPU
implementation, and 4 $\times$ higher energy efficiency than state-of-the-art
GPU accelerator optimized via CUBLAS GEMM.
