  This paper proposes a new hardware accelerator for sparse convolutional
neural networks (CNNs) by building a hardware unit to perform the Image to
Column (IM2COL) transformation of the input feature map coupled with a systolic
array-based general matrix-matrix multiplication (GEMM) unit. Our design
carefully overlaps the IM2COL transformation with the GEMM computation to
maximize parallelism. We propose a novel design for the IM2COL unit that uses a
set of distributed local memories connected by a ring network, which improves
energy efficiency and latency by streaming the input feature map only once. We
propose a tall systolic array for the GEMM unit while also providing the
ability to organize it as multiple small GEMM units, which enables our design
to handle a wide range of CNNs and their parameters. Further, our design
improves performance by effectively mapping the sparse data to the hardware
units by utilizing sparsity in both input feature maps and weights. Our
prototype, SPOTS, is on average 1.74X faster than Eyeriss. It is also 78X, and
12X more energy-efficient when compared to CPU and GPU implementations,
respectively.
