  In this work, we study the problem of continual learning (CL) where the goal
is to learn a model on a sequence of tasks, such that the data from the
previous tasks becomes unavailable while learning on the current task data. CL
is essentially a balancing act between being able to learn on the new task
(i.e., plasticity) and maintaining the performance on the previously learned
concepts (i.e., stability). Intending to address the stability-plasticity
trade-off, we propose to perform weight-ensembling of the model parameters of
the previous and current tasks. This weighted-ensembled model, which we call
Continual Model Averaging (or CoMA), attains high accuracy on the current task
by leveraging plasticity, while not deviating too far from the previous weight
configuration, ensuring stability. We also propose an improved variant of CoMA,
named Continual Fisher-weighted Model Averaging (or CoFiMA), that selectively
weighs each parameter in the weights ensemble by leveraging the Fisher
information of the weights of the model. Both variants are conceptually simple,
easy to implement, and effective in attaining state-of-the-art performance on
several standard CL benchmarks. Code is available at:
https://github.com/IemProg/CoFiMA.
