  Energy-based models parameterize the unnormalized log-probability of data
samples, but there is a lack of guidance on how to construct the "energy". In
this paper, we propose a Denoising-EBM which decomposes the image energy into
"semantic energy" and "texture energy". We define the "semantic energy" in the
latent space of DAE to model the high-level representations, and define the
pixel-level reconstruction error for denoising as "texture energy". Inspired by
score-based model, our model utilizes multi-scale noisy samples for
maximum-likelihood training and it outputs a vector instead of a scalar for
exploring a larger set of functions during optimization. After training, the
semantics are first synthesized by fast MCMC through "semantic energy", and
then the pixel-level refinement of semantic image will be performed to generate
perfect samples based on "texture energy". Ultimately, our model can outperform
most EBMs in image generation. And we also demonstrate that Denoising-EBM has
top performance among EBMs for out-of-distribution detection.
