  Can a text-to-image diffusion model be used as a training objective for
adapting a GAN generator to another domain? In this paper, we show that the
classifier-free guidance can be leveraged as a critic and enable generators to
distill knowledge from large-scale text-to-image diffusion models. Generators
can be efficiently shifted into new domains indicated by text prompts without
access to groundtruth samples from target domains. We demonstrate the
effectiveness and controllability of our method through extensive experiments.
Although not trained to minimize CLIP loss, our model achieves equally high
CLIP scores and significantly lower FID than prior work on short prompts, and
outperforms the baseline qualitatively and quantitatively on long and
complicated prompts. To our best knowledge, the proposed method is the first
attempt at incorporating large-scale pre-trained diffusion models and
distillation sampling for text-driven image generator domain adaptation and
gives a quality previously beyond possible. Moreover, we extend our work to
3D-aware style-based generators and DreamBooth guidance.
