  The goal of continual learning is to improve the performance of recognition
models in learning sequentially arrived data. Although most existing works are
established on the premise of learning from scratch, growing efforts have been
devoted to incorporating the benefits of pre-training. However, how to
adaptively exploit the pre-trained knowledge for each incremental task while
maintaining its generalizability remains an open question. In this work, we
present an extensive analysis for continual learning on a pre-trained model
(CLPM), and attribute the key challenge to a progressive overfitting problem.
Observing that selectively reducing the learning rate can almost resolve this
issue in the representation layer, we propose a simple but extremely effective
approach named Slow Learner with Classifier Alignment (SLCA), which further
improves the classification layer by modeling the class-wise distributions and
aligning the classification layers in a post-hoc fashion. Across a variety of
scenarios, our proposal provides substantial improvements for CLPM (e.g., up to
49.76%, 50.05%, 44.69% and 40.16% on Split CIFAR-100, Split ImageNet-R, Split
CUB-200 and Split Cars-196, respectively), and thus outperforms
state-of-the-art approaches by a large margin. Based on such a strong baseline,
critical factors and promising directions are analyzed in-depth to facilitate
subsequent research. Code has been made available at:
https://github.com/GengDavid/SLCA.
