  The increasing computational requirements of deep neural networks (DNNs) have
led to significant interest in obtaining DNN models that are sparse, yet
accurate. Recent work has investigated the even harder case of sparse training,
where the DNN weights are, for as much as possible, already sparse to reduce
computational costs during training. Existing sparse training methods are often
empirical and can have lower accuracy relative to the dense baseline. In this
paper, we present a general approach called Alternating Compressed/DeCompressed
(AC/DC) training of DNNs, demonstrate convergence for a variant of the
algorithm, and show that AC/DC outperforms existing sparse training methods in
accuracy at similar computational budgets; at high sparsity levels, AC/DC even
outperforms existing methods that rely on accurate pre-trained dense models. An
important property of AC/DC is that it allows co-training of dense and sparse
models, yielding accurate sparse-dense model pairs at the end of the training
process. This is useful in practice, where compressed variants may be desirable
for deployment in resource-constrained settings without re-doing the entire
training flow, and also provides us with insights into the accuracy gap between
dense and compressed models. The code is available at:
https://github.com/IST-DASLab/ACDC .
