  Existing work on continual learning (CL) is primarily devoted to developing
algorithms for models trained from scratch. Despite their encouraging
performance on contrived benchmarks, these algorithms show dramatic performance
drops in real-world scenarios. Therefore, this paper advocates the systematic
introduction of pre-training to CL, which is a general recipe for transferring
knowledge to downstream tasks but is substantially missing in the CL community.
Our investigation reveals the multifaceted complexity of exploiting pre-trained
models for CL, along three different axes, pre-trained models, CL algorithms,
and CL scenarios. Perhaps most intriguingly, improvements in CL algorithms from
pre-training are very inconsistent an underperforming algorithm could become
competitive and even state-of-the-art when all algorithms start from a
pre-trained model. This indicates that the current paradigm, where all CL
methods are compared in from-scratch training, is not well reflective of the
true CL objective and desired progress. In addition, we make several other
important observations, including that CL algorithms that exert less
regularization benefit more from a pre-trained model; and that a stronger
pre-trained model such as CLIP does not guarantee a better improvement. Based
on these findings, we introduce a simple yet effective baseline that employs
minimum regularization and leverages the more beneficial pre-trained model,
coupled with a two-stage training pipeline. We recommend including this strong
baseline in the future development of CL algorithms, due to its demonstrated
state-of-the-art performance.
