  We present a continual learning approach for generative adversarial networks
(GANs), by designing and leveraging parameter-efficient feature map
transformations. Our approach is based on learning a set of global and
task-specific parameters. The global parameters are fixed across tasks whereas
the task-specific parameters act as local adapters for each task, and help in
efficiently obtaining task-specific feature maps. Moreover, we propose an
element-wise addition of residual bias in the transformed feature space, which
further helps stabilize GAN training in such settings. Our approach also
leverages task similarity information based on the Fisher information matrix.
Leveraging this knowledge from previous tasks significantly improves the model
performance. In addition, the similarity measure also helps reduce the
parameter growth in continual adaptation and helps to learn a compact model. In
contrast to the recent approaches for continually-learned GANs, the proposed
approach provides a memory-efficient way to perform effective continual data
generation. Through extensive experiments on challenging and diverse datasets,
we show that the feature-map-transformation approach outperforms
state-of-the-art methods for continually-learned GANs, with substantially fewer
parameters. The proposed method generates high-quality samples that can also
improve the generative-replay-based continual learning for discriminative
tasks.
