  Does a Variational AutoEncoder (VAE) consistently encode typical samples
generated from its decoder? This paper shows that the perhaps surprising answer
to this question is `No'; a (nominally trained) VAE does not necessarily
amortize inference for typical samples that it is capable of generating. We
study the implications of this behaviour on the learned representations and
also the consequences of fixing it by introducing a notion of self consistency.
Our approach hinges on an alternative construction of the variational
approximation distribution to the true posterior of an extended VAE model with
a Markov chain alternating between the encoder and the decoder. The method can
be used to train a VAE model from scratch or given an already trained VAE, it
can be run as a post processing step in an entirely self supervised way without
access to the original training data. Our experimental analysis reveals that
encoders trained with our self-consistency approach lead to representations
that are robust (insensitive) to perturbations in the input introduced by
adversarial attacks. We provide experimental results on the ColorMnist and
CelebA benchmark datasets that quantify the properties of the learned
representations and compare the approach with a baseline that is specifically
trained for the desired property.
