  We address a challenging lifelong few-shot image generation task for the
first time. In this situation, a generative model learns a sequence of tasks
using only a few samples per task. Consequently, the learned model encounters
both catastrophic forgetting and overfitting problems at a time. Existing
studies on lifelong GANs have proposed modulation-based methods to prevent
catastrophic forgetting. However, they require considerable additional
parameters and cannot generate high-fidelity and diverse images from limited
data. On the other hand, the existing few-shot GANs suffer from severe
catastrophic forgetting when learning multiple tasks. To alleviate these
issues, we propose a framework called Lifelong Few-Shot GAN (LFS-GAN) that can
generate high-quality and diverse images in lifelong few-shot image generation
task. Our proposed framework learns each task using an efficient task-specific
modulator - Learnable Factorized Tensor (LeFT). LeFT is rank-constrained and
has a rich representation ability due to its unique reconstruction technique.
Furthermore, we propose a novel mode seeking loss to improve the diversity of
our model in low-data circumstances. Extensive experiments demonstrate that the
proposed LFS-GAN can generate high-fidelity and diverse images without any
forgetting and mode collapse in various domains, achieving state-of-the-art in
lifelong few-shot image generation task. Surprisingly, we find that our LFS-GAN
even outperforms the existing few-shot GANs in the few-shot image generation
task. The code is available at Github.
