  Deep learning has proved to be a successful paradigm for solving many
challenges in machine learning. However, deep neural networks fail when trained
sequentially on multiple tasks, a shortcoming known as catastrophic forgetting
in the continual learning literature. Despite a recent flourish of learning
algorithms successfully addressing this problem, we find that provable
guarantees against catastrophic forgetting are lacking. In this work, we study
the relationship between learning and forgetting by looking at the geometry of
neural networks' loss landscape. We offer a unifying perspective on a family of
continual learning algorithms, namely methods based on parameter isolation, and
we establish guarantees on catastrophic forgetting for some of them.
