  There has been tremendous progress in large-scale text-to-image synthesis
driven by diffusion models enabling versatile downstream applications such as
3D object synthesis from texts, image editing, and customized generation. We
present a generic approach using latent diffusion models as powerful image
priors for various visual synthesis tasks. Existing methods that utilize such
priors fail to use these models' full capabilities. To improve this, our core
ideas are 1) a feature matching loss between features from different layers of
the decoder to provide detailed guidance and 2) a KL divergence loss to
regularize the predicted latent features and stabilize the training. We
demonstrate the efficacy of our approach on three different applications,
text-to-3D, StyleGAN adaptation, and layered image editing. Extensive results
show our method compares favorably against baselines.
