  Standard artificial neural networks suffer from the well-known issue of
catastrophic forgetting, making continual or lifelong learning difficult for
machine learning. In recent years, numerous methods have been proposed for
continual learning, but due to differences in evaluation protocols it is
difficult to directly compare their performance. To enable more structured
comparisons, we describe three continual learning scenarios based on whether at
test time task identity is provided and--in case it is not--whether it must be
inferred. Any sequence of well-defined tasks can be performed according to each
scenario. Using the split and permuted MNIST task protocols, for each scenario
we carry out an extensive comparison of recently proposed continual learning
methods. We demonstrate substantial differences between the three scenarios in
terms of difficulty and in terms of how efficient different methods are. In
particular, when task identity must be inferred (i.e., class incremental
learning), we find that regularization-based approaches (e.g., elastic weight
consolidation) fail and that replaying representations of previous experiences
seems required for solving this scenario.
