  Continual learning refers to the ability to acquire and transfer knowledge
without catastrophically forgetting what was previously learned. In this work,
we consider \emph{few-shot} continual learning in classification tasks, and we
propose a novel method, Variational Prototype Replays, that efficiently
consolidates and recalls previous knowledge to avoid catastrophic forgetting.
In each classification task, our method learns a set of variational prototypes
with their means and variances, where embedding of the samples from the same
class can be represented in a prototypical distribution and
class-representative prototypes are separated apart. To alleviate catastrophic
forgetting, our method replays one sample per class from previous tasks, and
correspondingly matches newly predicted embeddings to their nearest
class-representative prototypes stored from previous tasks. Compared with
recent continual learning approaches, our method can readily adapt to new tasks
with more classes without requiring the addition of new units. Furthermore, our
method is more memory efficient since only class-representative prototypes with
their means and variances, as well as only one sample per class from previous
tasks need to be stored. Without tampering with the performance on initial
tasks, our method learns novel concepts given a few training examples of each
class in new tasks.
