  Recent breakthroughs in self-supervised learning show that such algorithms
learn visual representations that can be transferred better to unseen tasks
than joint-training methods relying on task-specific supervision. In this
paper, we found that the similar holds in the continual learning con-text:
contrastively learned representations are more robust against the catastrophic
forgetting than jointly trained representations. Based on this novel
observation, we propose a rehearsal-based continual learning algorithm that
focuses on continually learning and maintaining transferable representations.
More specifically, the proposed scheme (1) learns representations using the
contrastive learning objective, and (2) preserves learned representations using
a self-supervised distillation step. We conduct extensive experimental
validations under popular benchmark image classification datasets, where our
method sets the new state-of-the-art performance.
