  As neural networks become more popular, the need for accompanying uncertainty
estimates increases. There are currently two main approaches to test the
quality of these estimates. Most methods output a density. They can be compared
by evaluating their loglikelihood on a test set. Other methods output a
prediction interval directly. These methods are often tested by examining the
fraction of test points that fall inside the corresponding prediction
intervals. Intuitively both approaches seem logical. However, we demonstrate
through both theoretical arguments and simulations that both ways of evaluating
the quality of uncertainty estimates have serious flaws. Firstly, both
approaches cannot disentangle the separate components that jointly create the
predictive uncertainty, making it difficult to evaluate the quality of the
estimates of these components. Secondly, a better loglikelihood does not
guarantee better prediction intervals, which is what the methods are often used
for in practice. Moreover, the current approach to test prediction intervals
directly has additional flaws. We show why it is fundamentally flawed to test a
prediction or confidence interval on a single test set. At best, marginal
coverage is measured, implicitly averaging out overconfident and underconfident
predictions. A much more desirable property is pointwise coverage, requiring
the correct coverage for each prediction. We demonstrate through practical
examples that these effects can result in favoring a method, based on the
predictive uncertainty, that has undesirable behaviour of the confidence or
prediction intervals. Finally, we propose a simulation-based testing approach
that addresses these problems while still allowing easy comparison between
different methods.
