  An energy-based model (EBM) is a popular generative framework that offers
both explicit density and architectural flexibility, but training them is
difficult since it is often unstable and time-consuming. In recent years,
various training techniques have been developed, e.g., better divergence
measures or stabilization in MCMC sampling, but there often exists a large gap
between EBMs and other generative frameworks like GANs in terms of generation
quality. In this paper, we propose a novel and effective framework for
improving EBMs via contrastive representation learning (CRL). To be specific,
we consider representations learned by contrastive methods as the true
underlying latent variable. This contrastive latent variable could guide EBMs
to understand the data structure better, so it can improve and accelerate EBM
training significantly. To enable the joint training of EBM and CRL, we also
design a new class of latent-variable EBMs for learning the joint density of
data and the contrastive latent variable. Our experimental results demonstrate
that our scheme achieves lower FID scores, compared to prior-art EBM methods
(e.g., additionally using variational autoencoders or diffusion techniques),
even with significantly faster and more memory-efficient training. We also show
conditional and compositional generation abilities of our latent-variable EBMs
as their additional benefits, even without explicit conditional training. The
code is available at https://github.com/hankook/CLEL.
