  The compression of deep neural networks (DNNs) to reduce inference cost
becomes increasingly important to meet realistic deployment requirements of
various applications. There have been a significant amount of work regarding
network compression, while most of them are heuristic rule-based or typically
not friendly to be incorporated into varying scenarios. On the other hand,
sparse optimization yielding sparse solutions naturally fits the compression
requirement, but due to the limited study of sparse optimization in stochastic
learning, its extension and application onto model compression is rarely well
explored. In this work, we propose a model compression framework based on the
recent progress on sparse stochastic optimization. Compared to existing model
compression techniques, our method is effective and requires fewer extra
engineering efforts to incorporate with varying applications, and has been
numerically demonstrated on benchmark compression tasks. Particularly, we
achieve up to 7.2 and 2.9 times FLOPs reduction with the same level of
evaluation accuracy on VGG16 for CIFAR10 and ResNet50 for ImageNet compared to
the baseline heavy models, respectively.
