  In this work, we provide a deterministic alternative to the stochastic
variational training of generative autoencoders. We refer to these new
generative autoencoders as AutoEncoders within Flows (AEF), since the encoder
and decoder are defined as affine layers of an overall invertible architecture.
This results in a deterministic encoding of the data, as opposed to the
stochastic encoding of VAEs. The paper introduces two related families of AEFs.
The first family relies on a partition of the ambient space and is trained by
exact maximum-likelihood. The second family exploits a deterministic expansion
of the ambient space and is trained by maximizing the log-probability in this
extended space. This latter case leaves complete freedom in the choice of
encoder, decoder and prior architectures, making it a drop-in replacement for
the training of existing VAEs and VAE-style models. We show that these AEFs can
have strikingly higher performance than architecturally identical VAEs in terms
of log-likelihood and sample quality, especially for low dimensional latent
spaces. Importantly, we show that AEF samples are substantially sharper than
VAE samples.
