  We focus on the continual learning problem where the tasks arrive
sequentially and the aim is to perform well on the newly arrived task without
performance degradation on the previously seen tasks. In contrast to the
continual learning literature focusing on the centralized setting, we
investigate the distributed estimation framework. We consider the
well-established distributed learning algorithm COCOA. We derive closed form
expressions for the iterations for the overparametrized case. We illustrate the
convergence and the error performance of the algorithm based on the
over/under-parameterization of the problem. Our results show that depending on
the problem dimensions and data generation assumptions, COCOA can perform
continual learning over a sequence of tasks, i.e., it can learn a new task
without forgetting previously learned tasks, with access only to one task at a
time.
