  We introduce ensembles of stochastic neural networks to approximate the
Bayesian posterior, combining stochastic methods such as dropout with deep
ensembles. The stochastic ensembles are formulated as families of distributions
and trained to approximate the Bayesian posterior with variational inference.
We implement stochastic ensembles based on Monte Carlo dropout, DropConnect and
a novel non-parametric version of dropout and evaluate them on a toy problem
and CIFAR image classification. For both tasks, we test the quality of the
posteriors directly against Hamiltonian Monte Carlo simulations. Our results
show that stochastic ensembles provide more accurate posterior estimates than
other popular baselines for Bayesian inference.
