  Deep neural networks with lots of parameters are typically used for
large-scale computer vision tasks such as image classification. This is a
result of using dense matrix multiplications and convolutions. However, sparse
computations are known to be much more efficient. In this work, we train and
build neural networks which implicitly use sparse computations. We introduce
additional gate variables to perform parameter selection and show that this is
equivalent to using a spike-and-slab prior. We experimentally validate our
method on both small and large networks and achieve state-of-the-art
compression results for sparse neural network models.
