  Pruning large neural networks to create high-quality, independently trainable
sparse masks, which can maintain similar performance to their dense
counterparts, is very desirable due to the reduced space and time complexity.
As research effort is focused on increasingly sophisticated pruning methods
that leads to sparse subnetworks trainable from the scratch, we argue for an
orthogonal, under-explored theme: improving training techniques for pruned
sub-networks, i.e. sparse training. Apart from the popular belief that only the
quality of sparse masks matters for sparse training, in this paper we
demonstrate an alternative opportunity: one can carefully customize the sparse
training techniques to deviate from the default dense network training
protocols, consisting of introducing ``ghost" neurons and skip connections at
the early stage of training, and strategically modifying the initialization as
well as labels. Our new sparse training recipe is generally applicable to
improving training from scratch with various sparse masks. By adopting our
newly curated techniques, we demonstrate significant performance gains across
various popular datasets (CIFAR-10, CIFAR-100, TinyImageNet), architectures
(ResNet-18/32/104, Vgg16, MobileNet), and sparse mask options (lottery ticket,
SNIP/GRASP, SynFlow, or even randomly pruning), compared to the default
training protocols, especially at high sparsity levels. Code is at
https://github.com/VITA-Group/ToST
