  The continuous ranked probability score (crps) is the most commonly used
scoring rule in the evaluation of probabilistic forecasts for real-valued
outcomes. To assess and rank forecasting methods, researchers compute the mean
crps over given sets of forecast situations, based on the respective predictive
distributions and outcomes. We propose a new, isotonicity-based decomposition
of the mean crps into interpretable components that quantify miscalibration
(MSC), discrimination ability (DSC), and uncertainty (UNC), respectively. In a
detailed theoretical analysis, we compare the new approach to empirical
decompositions proposed earlier, generalize to population versions, analyse
their properties and relationships, and relate to a hierarchy of notions of
calibration. The isotonicity-based decomposition guarantees the nonnegativity
of the components and quantifies calibration in a sense that is stronger than
for other types of decompositions, subject to the nondegeneracy of empirical
decompositions. We illustrate the usage of the isotonicity-based decomposition
in case studies from weather prediction and machine learning.
