  Generative Adversarial Networks (GANs) rely heavily on large-scale training
data for training high-quality image generation models. With limited training
data, the GAN discriminator often suffers from severe overfitting which
directly leads to degraded generation especially in generation diversity.
Inspired by the recent advances in knowledge distillation (KD), we propose
KD-DLGAN, a knowledge-distillation based generation framework that introduces
pre-trained vision-language models for training effective data-limited
generation models. KD-DLGAN consists of two innovative designs. The first is
aggregated generative KD that mitigates the discriminator overfitting by
challenging the discriminator with harder learning tasks and distilling more
generalizable knowledge from the pre-trained models. The second is correlated
generative KD that improves the generation diversity by distilling and
preserving the diverse image-text correlation within the pre-trained models.
Extensive experiments over multiple benchmarks show that KD-DLGAN achieves
superior image generation with limited training data. In addition, KD-DLGAN
complements the state-of-the-art with consistent and substantial performance
gains.
