  In this paper, we propose conjugate energy-based models (CEBMs), a new class
of energy-based models that define a joint density over data and latent
variables. The joint density of a CEBM decomposes into an intractable
distribution over data and a tractable posterior over latent variables. CEBMs
have similar use cases as variational autoencoders, in the sense that they
learn an unsupervised mapping from data to latent variables. However, these
models omit a generator network, which allows them to learn more flexible
notions of similarity between data points. Our experiments demonstrate that
conjugate EBMs achieve competitive results in terms of image modelling,
predictive power of latent space, and out-of-domain detection on a variety of
datasets.
