  Deep neural networks are known to suffer the catastrophic forgetting problem,
where they tend to forget the knowledge from the previous tasks when
sequentially learning new tasks. Such failure hinders the application of deep
learning based vision system in continual learning settings. In this work, we
present a simple yet surprisingly effective way of preventing catastrophic
forgetting. Our method, called Few-shot Self Reminder (FSR), regularizes the
neural net from changing its learned behaviour by performing logit matching on
selected samples kept in episodic memory from the old tasks. Surprisingly, this
simplistic approach only requires to retrain a small amount of data in order to
outperform previous methods in knowledge retention. We demonstrate the
superiority of our method to the previous ones in two different continual
learning settings on popular benchmarks, as well as a new continual learning
problem where tasks are designed to be more dissimilar.
