  Variational autoencoders (VAEs) are popular models for representation
learning but their encoders are susceptible to overfitting (Cremer et al.,
2018) because they are trained on a finite training set instead of the true
(continuous) data distribution $p_{\mathrm{data}}(\mathbf{x})$. Diffusion
models, on the other hand, avoid this issue by keeping the encoder fixed. This
makes their representations less interpretable, but it simplifies training,
enabling accurate and continuous approximations of
$p_{\mathrm{data}}(\mathbf{x})$. In this paper, we show that overfitting
encoders in VAEs can be effectively mitigated by training on samples from a
pre-trained diffusion model. These results are somewhat unexpected as recent
findings (Alemohammad et al., 2023; Shumailov et al., 2023) observe a decay in
generative performance when models are trained on data generated by another
generative model. We analyze generalization performance, amortization gap, and
robustness of VAEs trained with our proposed method on three different data
sets. We find improvements in all metrics compared to both normal training and
conventional data augmentation methods, and we show that a modest amount of
samples from the diffusion model suffices to obtain these gains.
