  Kernel continual learning by \citet{derakhshani2021kernel} has recently
emerged as a strong continual learner due to its non-parametric ability to
tackle task interference and catastrophic forgetting. Unfortunately its success
comes at the expense of an explicit memory to store samples from past tasks,
which hampers scalability to continual learning settings with a large number of
tasks. In this paper, we introduce generative kernel continual learning, which
explores and exploits the synergies between generative models and kernels for
continual learning. The generative model is able to produce representative
samples for kernel learning, which removes the dependence on memory in kernel
continual learning. Moreover, as we replay only on the generative model, we
avoid task interference while being computationally more efficient compared to
previous methods that need replay on the entire model. We further introduce a
supervised contrastive regularization, which enables our model to generate even
more discriminative samples for better kernel-based classification performance.
We conduct extensive experiments on three widely-used continual learning
benchmarks that demonstrate the abilities and benefits of our contributions.
Most notably, on the challenging SplitCIFAR100 benchmark, with just a simple
linear kernel we obtain the same accuracy as kernel continual learning with
variational random features for one tenth of the memory, or a 10.1\% accuracy
gain for the same memory budget.
