  Reliably estimating the uncertainty of a prediction throughout the model
lifecycle is crucial in many safety-critical applications. The most common way
to measure this uncertainty is via the predicted confidence. While this tends
to work well for in-domain samples, these estimates are unreliable under domain
drift and restricted to classification. Alternatively, proper scores can be
used for most predictive tasks but a bias-variance decomposition for model
uncertainty does not exist in the current literature. In this work we introduce
a general bias-variance decomposition for proper scores, giving rise to the
Bregman Information as the variance term. We discover how exponential families
and the classification log-likelihood are special cases and provide novel
formulations. Surprisingly, we can express the classification case purely in
the logit space. We showcase the practical relevance of this decomposition on
several downstream tasks, including model ensembles and confidence regions.
Further, we demonstrate how different approximations of the instance-level
Bregman Information allow reliable out-of-distribution detection for all
degrees of domain drift.
