  Dynamic Sparse Training (DST) methods achieve state-of-the-art results in
sparse neural network training, matching the generalization of dense models
while enabling sparse training and inference. Although the resulting models are
highly sparse and theoretically less computationally expensive, achieving
speedups with unstructured sparsity on real-world hardware is challenging. In
this work, we propose a sparse-to-sparse DST method, Structured RigL (SRigL),
to learn a variant of fine-grained structured N:M sparsity by imposing a
constant fan-in constraint. Using our empirical analysis of existing DST
methods at high sparsity, we additionally employ a neuron ablation method which
enables SRigL to achieve state-of-the-art sparse-to-sparse structured DST
performance on a variety of Neural Network (NN) architectures. Using a 90%
sparse linear layer, we demonstrate a real-world acceleration of 3.4x/2.5x on
CPU for online inference and 1.7x/13.0x on GPU for inference with a batch size
of 256 when compared to equivalent dense/unstructured (CSR) sparse layers,
respectively.
