  This paper describes a novel approach of packing sparse convolutional neural
networks for their efficient systolic array implementations. By combining
subsets of columns in the original filter matrix associated with a
convolutional layer, we increase the utilization efficiency of the systolic
array substantially (e.g., ~4x) due to the increased density of nonzeros in the
resulting packed filter matrix. In combining columns, for each row, all filter
weights but one with the largest magnitude are pruned. We retrain the remaining
weights to preserve high accuracy. We demonstrate that in mitigating data
privacy concerns the retraining can be accomplished with only fractions of the
original dataset (e.g., 10\% for CIFAR-10). We study the effectiveness of this
joint optimization for both high utilization and classification accuracy with
ASIC and FPGA designs based on efficient bit-serial implementations of
multiplier-accumulators. We present analysis and empirical evidence on the
superior performance of our column combining approach against prior arts under
metrics such as energy efficiency (3x) and inference latency (12x).
