  The recent developments in the machine learning domain have enabled the
development of complex multivariate probabilistic forecasting models.
Therefore, it is pivotal to have a precise evaluation method to gauge the
performance and predictability power of these complex methods. To do so,
several evaluation metrics have been proposed in the past (such as Energy
Score, Dawid-Sebastiani score, variogram score), however, they cannot reliably
measure the performance of a probabilistic forecaster. Recently, CRPS-sum has
gained a lot of prominence as a reliable metric for multivariate probabilistic
forecasting. This paper presents a systematic evaluation of CRPS-sum to
understand its discrimination ability. We show that the statistical properties
of target data affect the discrimination ability of CRPS-Sum. Furthermore, we
highlight that CRPS-Sum calculation overlooks the performance of the model on
each dimension. These flaws can lead us to an incorrect assessment of model
performance. Finally, with experiments on the real-world dataset, we
demonstrate that the shortcomings of CRPS-Sum provide a misleading indication
of the probabilistic forecasting performance method. We show that it is easily
possible to have a better CRPS-Sum for a dummy model, which looks like random
noise, in comparison to the state-of-the-art method.
