  In this work, we introduce Adapt & Align, a method for continual learning of
neural networks by aligning latent representations in generative models. Neural
Networks suffer from abrupt loss in performance when retrained with additional
training data from different distributions. At the same time, training with
additional data without access to the previous examples rarely improves the
model's performance. In this work, we propose a new method that mitigates those
problems by employing generative models and splitting the process of their
update into two parts. In the first one, we train a local generative model
using only data from a new task. In the second phase, we consolidate latent
representations from the local model with a global one that encodes knowledge
of all past experiences. We introduce our approach with Variational
Auteoncoders and Generative Adversarial Networks. Moreover, we show how we can
use those generative models as a general method for continual knowledge
consolidation that can be used in downstream tasks such as classification.
