  Modern generative models are usually designed to match target distributions
directly in the data space, where the intrinsic dimension of data can be much
lower than the ambient dimension. We argue that this discrepancy may contribute
to the difficulties in training generative models. We therefore propose to map
both the generated and target distributions to a latent space using the encoder
of a standard autoencoder, and train the generator (or decoder) to match the
target distribution in the latent space. Specifically, we enforce the
consistency in both the data space and the latent space with theoretically
justified data and latent reconstruction losses. The resulting generative
model, which we call a perceptual generative autoencoder (PGA), is then trained
with a maximum likelihood or variational autoencoder (VAE) objective. With
maximum likelihood, PGAs generalize the idea of reversible generative models to
unrestricted neural network architectures and arbitrary number of latent
dimensions. When combined with VAEs, PGAs substantially improve over the
baseline VAEs in terms of sample quality. Compared to other autoencoder-based
generative models using simple priors, PGAs achieve state-of-the-art FID scores
on CIFAR-10 and CelebA.
