  Continual learning methods are known to suffer from catastrophic forgetting,
a phenomenon that is particularly hard to counter for methods that do not store
exemplars of previous tasks. Therefore, to reduce potential drift in the
feature extractor, existing exemplar-free methods are typically evaluated in
settings where the first task is significantly larger than subsequent tasks.
Their performance drops drastically in more challenging settings starting with
a smaller first task. To address this problem of feature drift estimation for
exemplar-free methods, we propose to adversarially perturb the current samples
such that their embeddings are close to the old class prototypes in the old
model embedding space. We then estimate the drift in the embedding space from
the old to the new model using the perturbed images and compensate the
prototypes accordingly. We exploit the fact that adversarial samples are
transferable from the old to the new feature space in a continual learning
setting. The generation of these images is simple and computationally cheap. We
demonstrate in our experiments that the proposed approach better tracks the
movement of prototypes in embedding space and outperforms existing methods on
several standard continual learning benchmarks as well as on fine-grained
datasets. Code is available at https://github.com/dipamgoswami/ADC.
