  Deep convolutional neural networks (CNNs) are powerful tools for a wide range
of vision tasks, but the enormous amount of memory and compute resources
required by CNNs pose a challenge in deploying them on constrained devices.
Existing compression techniques, while excelling at reducing model sizes,
struggle to be computationally friendly. In this paper, we attend to the
statistical properties of sparse CNNs and present focused quantization, a novel
quantization strategy based on power-of-two values, which exploits the weight
distributions after fine-grained pruning. The proposed method dynamically
discovers the most effective numerical representation for weights in layers
with varying sparsities, significantly reducing model sizes. Multiplications in
quantized CNNs are replaced with much cheaper bit-shift operations for
efficient inference. Coupled with lossless encoding, we built a compression
pipeline that provides CNNs with high compression ratios (CR), low computation
cost and minimal loss in accuracy. In ResNet-50, we achieved a 18.08x CR with
only 0.24% loss in top-5 accuracy, outperforming existing compression methods.
We fully compressed a ResNet-18 and found that it is not only higher in CR and
top-5 accuracy, but also more hardware efficient as it requires fewer logic
gates to implement when compared to other state-of-the-art quantization methods
assuming the same throughput.
