  Deep generative models (DGMs) are data-eager because learning a complex model
on limited data suffers from a large variance and easily overfits. Inspired by
the classical perspective of the bias-variance tradeoff, we propose regularized
deep generative model (Reg-DGM), which leverages a nontransferable pre-trained
model to reduce the variance of generative modeling with limited data.
Formally, Reg-DGM optimizes a weighted sum of a certain divergence and the
expectation of an energy function, where the divergence is between the data and
the model distributions, and the energy function is defined by the pre-trained
model w.r.t. the model distribution. We analyze a simple yet representative
Gaussian-fitting case to demonstrate how the weighting hyperparameter trades
off the bias and the variance. Theoretically, we characterize the existence and
the uniqueness of the global minimum of Reg-DGM in a non-parametric setting and
prove its convergence with neural networks trained by gradient-based methods.
Empirically, with various pre-trained feature extractors and a data-dependent
energy function, Reg-DGM consistently improves the generation performance of
strong DGMs with limited data and achieves competitive results to the
state-of-the-art methods. Our implementation is available at
https://github.com/ML-GSAI/Reg-ADA-APA.
