  Deep ensembles have been empirically shown to be a promising approach for
improving accuracy, uncertainty and out-of-distribution robustness of deep
learning models. While deep ensembles were theoretically motivated by the
bootstrap, non-bootstrap ensembles trained with just random initialization also
perform well in practice, which suggests that there could be other explanations
for why deep ensembles work well. Bayesian neural networks, which learn
distributions over the parameters of the network, are theoretically
well-motivated by Bayesian principles, but do not perform as well as deep
ensembles in practice, particularly under dataset shift. One possible
explanation for this gap between theory and practice is that popular scalable
variational Bayesian methods tend to focus on a single mode, whereas deep
ensembles tend to explore diverse modes in function space. We investigate this
hypothesis by building on recent work on understanding the loss landscape of
neural networks and adding our own exploration to measure the similarity of
functions in the space of predictions. Our results show that random
initializations explore entirely different modes, while functions along an
optimization trajectory or sampled from the subspace thereof cluster within a
single mode predictions-wise, while often deviating significantly in the weight
space. Developing the concept of the diversity--accuracy plane, we show that
the decorrelation power of random initializations is unmatched by popular
subspace sampling methods. Finally, we evaluate the relative effects of
ensembling, subspace based methods and ensembles of subspace based methods, and
the experimental results validate our hypothesis.
