  Recently, sparse training has emerged as a promising paradigm for efficient
deep learning on edge devices. The current research mainly devotes efforts to
reducing training costs by further increasing model sparsity. However,
increasing sparsity is not always ideal since it will inevitably introduce
severe accuracy degradation at an extremely high sparsity level. This paper
intends to explore other possible directions to effectively and efficiently
reduce sparse training costs while preserving accuracy. To this end, we
investigate two techniques, namely, layer freezing and data sieving. First, the
layer freezing approach has shown its success in dense model training and
fine-tuning, yet it has never been adopted in the sparse training domain.
Nevertheless, the unique characteristics of sparse training may hinder the
incorporation of layer freezing techniques. Therefore, we analyze the
feasibility and potentiality of using the layer freezing technique in sparse
training and find it has the potential to save considerable training costs.
Second, we propose a data sieving method for dataset-efficient training, which
further reduces training costs by ensuring only a partial dataset is used
throughout the entire training process. We show that both techniques can be
well incorporated into the sparse training algorithm to form a generic
framework, which we dub SpFDE. Our extensive experiments demonstrate that SpFDE
can significantly reduce training costs while preserving accuracy from three
dimensions: weight sparsity, layer freezing, and dataset sieving.
