  A primary focus area in continual learning research is alleviating the
"catastrophic forgetting" problem in neural networks by designing new
algorithms that are more robust to the distribution shifts. While the recent
progress in continual learning literature is encouraging, our understanding of
what properties of neural networks contribute to catastrophic forgetting is
still limited. To address this, instead of focusing on continual learning
algorithms, in this work, we focus on the model itself and study the impact of
"width" of the neural network architecture on catastrophic forgetting, and show
that width has a surprisingly significant effect on forgetting. To explain this
effect, we study the learning dynamics of the network from various perspectives
such as gradient orthogonality, sparsity, and lazy training regime. We provide
potential explanations that are consistent with the empirical results across
different architectures and continual learning benchmarks.
