  While deep neural networks have become the go-to approach in computer vision,
the vast majority of these models fail to properly capture the uncertainty
inherent in their predictions. Estimating this predictive uncertainty can be
crucial, for example in automotive applications. In Bayesian deep learning,
predictive uncertainty is commonly decomposed into the distinct types of
aleatoric and epistemic uncertainty. The former can be estimated by letting a
neural network output the parameters of a certain probability distribution.
Epistemic uncertainty estimation is a more challenging problem, and while
different scalable methods recently have emerged, no extensive comparison has
been performed in a real-world setting. We therefore accept this task and
propose a comprehensive evaluation framework for scalable epistemic uncertainty
estimation methods in deep learning. Our proposed framework is specifically
designed to test the robustness required in real-world computer vision
applications. We also apply this framework to provide the first properly
extensive and conclusive comparison of the two current state-of-the-art
scalable methods: ensembling and MC-dropout. Our comparison demonstrates that
ensembling consistently provides more reliable and practically useful
uncertainty estimates. Code is available at
https://github.com/fregu856/evaluating_bdl.
