  We claim that a source of severe failures for Variational Auto-Encoders is
the choice of the distribution class used for the observation model.A first
theoretical and experimental contribution of the paper is to establish that
even in the large sample limit with arbitrarily powerful neural architectures
and latent space, the VAE failsif the sharpness of the distribution class does
not match the scale of the data.Our second claim is that the distribution
sharpness must preferably be learned by the VAE (as opposed to, fixed and
optimized offline): Autonomously adjusting this sharpness allows the VAE to
dynamically control the trade-off between the optimization of the
reconstruction loss and the latent compression. A second empirical contribution
is to show how the control of this trade-off is instrumental in escaping poor
local optima, akin a simulated annealing schedule.Both claims are backed upon
experiments on artificial data, MNIST and CelebA, showing how sharpness
learning addresses the notorious VAE blurriness issue.
