  In recent years, there has been a flurry of research in deep neural network
pruning and compression. Early approaches prune weights individually. However,
it is difficult to take advantage of the resulting unstructured sparsity
patterns on modern hardware like GPUs. As a result, pruning strategies which
impose sparsity structures in the weights have become more popular.
However,these structured pruning approaches typically lead to higher losses in
accuracy than unstructured pruning. In this paper, we present SparseRT, a code
generator that leverage unstructured sparsity to accelerate sparse linear
algebra operations in deep learning inference on GPUs. For 1x1 convolutions and
fully connected layers, we demonstrate geometric mean of speedups of 3.4x over
the equivalent dense computation at 90% sparsity and 5.4x at 95% sparsity when
evaluated on hundreds of test cases in deep learning. For sparse 3x3
convolutions, we show speedups of over 5x on use cases in ResNet-50.
