  Designing hardware accelerators for deep neural networks (DNNs) has been much
desired. Nonetheless, most of these existing accelerators are built for either
convolutional neural networks (CNNs) or recurrent neural networks (RNNs).
Recently, the Transformer model is replacing the RNN in the natural language
processing (NLP) area. However, because of intensive matrix computations and
complicated data flow being involved, the hardware design for the Transformer
model has never been reported. In this paper, we propose the first hardware
accelerator for two key components, i.e., the multi-head attention (MHA)
ResBlock and the position-wise feed-forward network (FFN) ResBlock, which are
the two most complex layers in the Transformer. Firstly, an efficient method is
introduced to partition the huge matrices in the Transformer, allowing the two
ResBlocks to share most of the hardware resources. Secondly, the computation
flow is well designed to ensure the high hardware utilization of the systolic
array, which is the biggest module in our design. Thirdly, complicated
nonlinear functions are highly optimized to further reduce the hardware
complexity and also the latency of the entire system. Our design is coded using
hardware description language (HDL) and evaluated on a Xilinx FPGA. Compared
with the implementation on GPU with the same setting, the proposed design
demonstrates a speed-up of 14.6x in the MHA ResBlock, and 3.4x in the FFN
ResBlock, respectively. Therefore, this work lays a good foundation for
building efficient hardware accelerators for multiple Transformer networks.
