  This paper studies the fundamental problem of learning multi-layer generator
models. The multi-layer generator model builds multiple layers of latent
variables as a prior model on top of the generator, which benefits learning
complex data distribution and hierarchical representations. However, such a
prior model usually focuses on modeling inter-layer relations between latent
variables by assuming non-informative (conditional) Gaussian distributions,
which can be limited in model expressivity. To tackle this issue and learn more
expressive prior models, we propose an energy-based model (EBM) on the joint
latent space over all layers of latent variables with the multi-layer generator
as its backbone. Such joint latent space EBM prior model captures the
intra-layer contextual relations at each layer through layer-wise energy terms,
and latent variables across different layers are jointly corrected. We develop
a joint training scheme via maximum likelihood estimation (MLE), which involves
Markov Chain Monte Carlo (MCMC) sampling for both prior and posterior
distributions of the latent variables from different layers. To ensure
efficient inference and learning, we further propose a variational training
scheme where an inference model is used to amortize the costly posterior MCMC
sampling. Our experiments demonstrate that the learned model can be expressive
in generating high-quality images and capturing hierarchical features for
better outlier detection.
