  Recent deep generative models (DGMs) such as generative adversarial networks
(GANs) and diffusion probabilistic models (DPMs) have shown their impressive
ability in generating high-fidelity photorealistic images. Although looking
appealing to human eyes, training a model on purely synthetic images for
downstream image processing tasks like image classification often results in an
undesired performance drop compared to training on real data. Previous works
have demonstrated that enhancing a real dataset with synthetic images from DGMs
can be beneficial. However, the improvements were subjected to certain
circumstances and yet were not comparable to adding the same number of real
images. In this work, we propose a new taxonomy to describe factors
contributing to this commonly observed phenomenon and investigate it on the
popular CIFAR-10 dataset. We hypothesize that the Content Gap accounts for a
large portion of the performance drop when using synthetic images from DGM and
propose strategies to better utilize them in downstream tasks. Extensive
experiments on multiple datasets showcase that our method outperforms baselines
on downstream classification tasks both in case of training on synthetic only
(Synthetic-to-Real) and training on a mix of real and synthetic data (Data
Augmentation), particularly in the data-scarce scenario.
