  This paper presents Systolic-CNN, an OpenCL-defined scalable,
run-time-flexible FPGA accelerator architecture, optimized for accelerating the
inference of various convolutional neural networks (CNNs) in multi-tenancy
cloud/edge computing. The existing OpenCL-defined FPGA accelerators for CNN
inference are insufficient due to limited flexibility for supporting multiple
CNN models at run time and poor scalability resulting in underutilized FPGA
resources and limited computational parallelism. Systolic-CNN adopts a highly
pipelined and paralleled 1-D systolic array architecture, which efficiently
explores both spatial and temporal parallelism for accelerating CNN inference
on FPGAs. Systolic-CNN is highly scalable and parameterized, which can be
easily adapted by users to achieve up to 100% utilization of the coarse-grained
computation resources (i.e., DSP blocks) for a given FPGA. Systolic-CNN is also
run-time-flexible in the context of multi-tenancy cloud/edge computing, which
can be time-shared to accelerate a variety of CNN models at run time without
the need of recompiling the FPGA kernel hardware nor reprogramming the FPGA.
The experiment results based on an Intel Arria/Stratix 10 GX FPGA Development
board show that the optimized single-precision implementation of Systolic-CNN
can achieve an average inference latency of 7ms/2ms, 84ms/33ms, 202ms/73ms,
1615ms/873ms, and 900ms/498ms per image for accelerating AlexNet, ResNet-50,
ResNet-152, RetinaNet, and Light-weight RetinaNet, respectively. Codes are
available at https://github.com/PSCLab-ASU/Systolic-CNN.
