  Continual learning (CL), which aims to learn a sequence of tasks, has
attracted significant recent attention. However, most work has focused on the
experimental performance of CL, and theoretical studies of CL are still
limited. In particular, there is a lack of understanding on what factors are
important and how they affect "catastrophic forgetting" and generalization
performance. To fill this gap, our theoretical analysis, under
overparameterized linear models, provides the first-known explicit form of the
expected forgetting and generalization error. Further analysis of such a key
result yields a number of theoretical explanations about how
overparameterization, task similarity, and task ordering affect both forgetting
and generalization error of CL. More interestingly, by conducting experiments
on real datasets using deep neural networks (DNNs), we show that some of these
insights even go beyond the linear models and can be carried over to practical
setups. In particular, we use concrete examples to show that our results not
only explain some interesting empirical observations in recent studies, but
also motivate better practical algorithm designs of CL.
