  While energy-based models (EBMs) exhibit a number of desirable properties,
training and sampling on high-dimensional datasets remains challenging.
Inspired by recent progress on diffusion probabilistic models, we present a
diffusion recovery likelihood method to tractably learn and sample from a
sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM
is trained with recovery likelihood, which maximizes the conditional
probability of the data at a certain noise level given their noisy versions at
a higher noise level. Optimizing recovery likelihood is more tractable than
marginal likelihood, as sampling from the conditional distributions is much
easier than sampling from the marginal distributions. After training,
synthesized images can be generated by the sampling process that initializes
from Gaussian white noise distribution and progressively samples the
conditional distributions at decreasingly lower noise levels. Our method
generates high fidelity samples on various image datasets. On unconditional
CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the
majority of GANs. Moreover, we demonstrate that unlike previous work on EBMs,
our long-run MCMC samples from the conditional distributions do not diverge and
still represent realistic images, allowing us to accurately estimate the
normalized density of data even for high-dimensional datasets. Our
implementation is available at https://github.com/ruiqigao/recovery_likelihood.
