  This paper introduces kernel continual learning, a simple but effective
variant of continual learning that leverages the non-parametric nature of
kernel methods to tackle catastrophic forgetting. We deploy an episodic memory
unit that stores a subset of samples for each task to learn task-specific
classifiers based on kernel ridge regression. This does not require memory
replay and systematically avoids task interference in the classifiers. We
further introduce variational random features to learn a data-driven kernel for
each task. To do so, we formulate kernel continual learning as a variational
inference problem, where a random Fourier basis is incorporated as the latent
variable. The variational posterior distribution over the random Fourier basis
is inferred from the coreset of each task. In this way, we are able to generate
more informative kernels specific to each task, and, more importantly, the
coreset size can be reduced to achieve more compact memory, resulting in more
efficient continual learning based on episodic memory. Extensive evaluation on
four benchmarks demonstrates the effectiveness and promise of kernels for
continual learning.
