  Generative foundation models like Stable Diffusion comprise a diverse
spectrum of knowledge in computer vision with the potential for transfer
learning, e.g., via generating data to train student models for downstream
tasks. This could circumvent the necessity of collecting labeled real-world
data, thereby presenting a form of data-free knowledge distillation. However,
the resultant student models show a significant drop in accuracy compared to
models trained on real data. We investigate possible causes for this drop and
focus on the role of the different layers of the student model. By training
these layers using either real or synthetic data, we reveal that the drop
mainly stems from the model's final layers. Further, we briefly investigate
other factors, such as differences in data-normalization between synthetic and
real, the impact of data augmentations, texture vs.\ shape learning, and
assuming oracle prompts. While we find that some of those factors can have an
impact, they are not sufficient to close the gap towards real data. Building
upon our insights that mainly later layers are responsible for the drop, we
investigate the data-efficiency of fine-tuning a synthetically trained model
with real data applied to only those last layers. Our results suggest an
improved trade-off between the amount of real training data used and the
model's accuracy. Our findings contribute to the understanding of the gap
between synthetic and real data and indicate solutions to mitigate the scarcity
of labeled real data.
