  Many applications require sparse neural networks due to space or inference
time restrictions. There is a large body of work on training dense networks to
yield sparse networks for inference, but this limits the size of the largest
trainable sparse model to that of the largest trainable dense model. In this
paper we introduce a method to train sparse neural networks with a fixed
parameter count and a fixed computational cost throughout training, without
sacrificing accuracy relative to existing dense-to-sparse training methods. Our
method updates the topology of the sparse network during training by using
parameter magnitudes and infrequent gradient calculations. We show that this
approach requires fewer floating-point operations (FLOPs) to achieve a given
level of accuracy compared to prior techniques. We demonstrate state-of-the-art
sparse training results on a variety of networks and datasets, including
ResNet-50, MobileNets on Imagenet-2012, and RNNs on WikiText-103. Finally, we
provide some insights into why allowing the topology to change during the
optimization can overcome local minima encountered when the topology remains
static. Code used in our work can be found in github.com/google-research/rigl.
