  Although variational autoencoders (VAEs) represent a widely influential deep
generative model, many aspects of the underlying energy function remain poorly
understood. In particular, it is commonly believed that Gaussian
encoder/decoder assumptions reduce the effectiveness of VAEs in generating
realistic samples. In this regard, we rigorously analyze the VAE objective,
differentiating situations where this belief is and is not actually true. We
then leverage the corresponding insights to develop a simple VAE enhancement
that requires no additional hyperparameters or sensitive tuning.
Quantitatively, this proposal produces crisp samples and stable FID scores that
are actually competitive with a variety of GAN models, all while retaining
desirable attributes of the original VAE architecture. A shorter version of
this work will appear in the ICLR 2019 conference proceedings (Dai and Wipf,
2019). The code for our model is available at https://github.com/daib13/
TwoStageVAE.
