  Previous works on sequential learning address the problem of forgetting in
discriminative models. In this paper we consider the case of generative models.
In particular, we investigate generative adversarial networks (GANs) in the
task of learning new categories in a sequential fashion. We first show that
sequential fine tuning renders the network unable to properly generate images
from previous categories (i.e. forgetting). Addressing this problem, we propose
Memory Replay GANs (MeRGANs), a conditional GAN framework that integrates a
memory replay generator. We study two methods to prevent forgetting by
leveraging these replays, namely joint training with replay and replay
alignment. Qualitative and quantitative experimental results in MNIST, SVHN and
LSUN datasets show that our memory replay approach can generate competitive
images while significantly mitigating the forgetting of previous categories.
