  $\textit{RigL}$, a sparse training algorithm, claims to directly train sparse
networks that match or exceed the performance of existing dense-to-sparse
training techniques (such as pruning) for a fixed parameter count and compute
budget. We implement $\textit{RigL}$ from scratch in Pytorch and reproduce its
performance on CIFAR-10 within 0.1% of the reported value. On both
CIFAR-10/100, the central claim holds -- given a fixed training budget,
$\textit{RigL}$ surpasses existing dynamic-sparse training methods over a range
of target sparsities. By training longer, the performance can match or exceed
iterative pruning, while consuming constant FLOPs throughout training. We also
show that there is little benefit in tuning $\textit{RigL}$'s hyper-parameters
for every sparsity, initialization pair -- the reference choice of
hyperparameters is often close to optimal performance. Going beyond the
original paper, we find that the optimal initialization scheme depends on the
training constraint. While the Erdos-Renyi-Kernel distribution outperforms the
Uniform distribution for a fixed parameter count, for a fixed FLOP count, the
latter performs better. Finally, redistributing layer-wise sparsity while
training can bridge the performance gap between the two initialization schemes,
but increases computational cost.
