  Variational autoencoders (VAEs) provide an effective and simple method for
modeling complex distributions. However, training VAEs often requires
considerable hyperparameter tuning to determine the optimal amount of
information retained by the latent variable. We study the impact of calibrated
decoders, which learn the uncertainty of the decoding distribution and can
determine this amount of information automatically, on the VAE performance.
While many methods for learning calibrated decoders have been proposed, many of
the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc
modifications instead. We perform the first comprehensive comparative analysis
of calibrated decoder and provide recommendations for simple and effective VAE
training. Our analysis covers a range of image and video datasets and several
single-image and sequential VAE models. We further propose a simple but novel
modification to the commonly used Gaussian decoder, which computes the
prediction variance analytically. We observe empirically that using heuristic
modifications is not necessary with our method. Project website is at
https://orybkin.github.io/sigma-vae/
