  Exemplar-free class-incremental learning using a backbone trained from
scratch and starting from a small first task presents a significant challenge
for continual representation learning. Prototype-based approaches, when
continually updated, face the critical issue of semantic drift due to which the
old class prototypes drift to different positions in the new feature space.
Through an analysis of prototype-based continual learning, we show that
forgetting is not due to diminished discriminative power of the feature
extractor, and can potentially be corrected by drift compensation. To address
this, we propose Learnable Drift Compensation (LDC), which can effectively
mitigate drift in any moving backbone, whether supervised or unsupervised. LDC
is fast and straightforward to integrate on top of existing continual learning
approaches. Furthermore, we showcase how LDC can be applied in combination with
self-supervised CL methods, resulting in the first exemplar-free
semi-supervised continual learning approach. We achieve state-of-the-art
performance in both supervised and semi-supervised settings across multiple
datasets. Code is available at \url{https://github.com/alviur/ldc}.
