  Continual learning, focused on sequentially learning multiple tasks, has
gained significant attention recently. Despite the tremendous progress made in
the past, the theoretical understanding, especially factors contributing to
catastrophic forgetting, remains relatively unexplored. In this paper, we
provide a general theoretical analysis of forgetting in the linear regression
model via Stochastic Gradient Descent (SGD) applicable to both
underparameterized and overparameterized regimes. Our theoretical framework
reveals some interesting insights into the intricate relationship between task
sequence and algorithmic parameters, an aspect not fully captured in previous
studies due to their restrictive assumptions. Specifically, we demonstrate
that, given a sufficiently large data size, the arrangement of tasks in a
sequence, where tasks with larger eigenvalues in their population data
covariance matrices are trained later, tends to result in increased forgetting.
Additionally, our findings highlight that an appropriate choice of step size
will help mitigate forgetting in both underparameterized and overparameterized
settings. To validate our theoretical analysis, we conducted simulation
experiments on both linear regression models and Deep Neural Networks (DNNs).
Results from these simulations substantiate our theoretical findings.
