  Existing work in continual learning (CL) focuses on mitigating catastrophic
forgetting, i.e., model performance deterioration on past tasks when learning a
new task. However, the training efficiency of a CL system is
under-investigated, which limits the real-world application of CL systems under
resource-limited scenarios. In this work, we propose a novel framework called
Sparse Continual Learning(SparCL), which is the first study that leverages
sparsity to enable cost-effective continual learning on edge devices. SparCL
achieves both training acceleration and accuracy preservation through the
synergy of three aspects: weight sparsity, data efficiency, and gradient
sparsity. Specifically, we propose task-aware dynamic masking (TDM) to learn a
sparse network throughout the entire CL process, dynamic data removal (DDR) to
remove less informative training data, and dynamic gradient masking (DGM) to
sparsify the gradient updates. Each of them not only improves efficiency, but
also further mitigates catastrophic forgetting. SparCL consistently improves
the training efficiency of existing state-of-the-art (SOTA) CL methods by at
most 23X less training FLOPs, and, surprisingly, further improves the SOTA
accuracy by at most 1.7%. SparCL also outperforms competitive baselines
obtained from adapting SOTA sparse training methods to the CL setting in both
efficiency and accuracy. We also evaluate the effectiveness of SparCL on a real
mobile phone, further indicating the practical potential of our method.
