  High demand for computation resources severely hinders deployment of
large-scale Deep Neural Networks (DNN) in resource constrained devices. In this
work, we propose a Structured Sparsity Learning (SSL) method to regularize the
structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs.
SSL can: (1) learn a compact structure from a bigger DNN to reduce computation
cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently
accelerate the DNNs evaluation. Experimental results show that SSL achieves on
average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet
against CPU and GPU, respectively, with off-the-shelf libraries. These speedups
are about twice speedups of non-structured sparsity; (3) regularize the DNN
structure to improve classification accuracy. The results show that for
CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual
Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%,
which is still slightly higher than that of original ResNet with 32 layers. For
AlexNet, structure regularization by SSL also reduces the error by around ~1%.
Open source code is in https://github.com/wenwei202/caffe/tree/scnn
