  Convolutional Neural Networks (CNNs) are the state-of-the-art solution for
many deep learning applications. For maximum scalability, their computation
should combine high performance and energy efficiency. In practice, the
convolutions of each CNN layer are mapped to a matrix multiplication that
includes all input features and kernels of each layer and is computed using a
systolic array. In this work, we focus on the design of a systolic array with
configurable pipeline with the goal to select an optimal pipeline configuration
for each CNN layer. The proposed systolic array, called ArrayFlex, can operate
in normal, or in shallow pipeline mode, thus balancing the execution time in
cycles and the operating clock frequency. By selecting the appropriate pipeline
configuration per CNN layer, ArrayFlex reduces the inference latency of
state-of-the-art CNNs by 11%, on average, as compared to a traditional
fixed-pipeline systolic array. Most importantly, this result is achieved while
using 13%-23% less power, for the same applications, thus offering a combined
energy-delay-product efficiency between 1.4x and 1.8x.
