  In the field of continual learning, models are designed to learn tasks one
after the other. While most research has centered on supervised continual
learning, there is a growing interest in unsupervised continual learning, which
makes use of the vast amounts of unlabeled data. Recent studies have
highlighted the strengths of unsupervised methods, particularly self-supervised
learning, in providing robust representations. The improved transferability of
those representations built with self-supervised methods is often associated
with the role played by the multi-layer perceptron projector. In this work, we
depart from this observation and reexamine the role of supervision in continual
representation learning. We reckon that additional information, such as human
annotations, should not deteriorate the quality of representations. Our
findings show that supervised models when enhanced with a multi-layer
perceptron head, can outperform self-supervised models in continual
representation learning. This highlights the importance of the multi-layer
perceptron projector in shaping feature transferability across a sequence of
tasks in continual learning. The code is available on github:
https://github.com/danielm1405/sl-vs-ssl-cl.
