  Multivariate probabilistic time series forecasts are commonly evaluated via
proper scoring rules, i.e., functions that are minimal in expectation for the
ground-truth distribution. However, this property is not sufficient to
guarantee good discrimination in the non-asymptotic regime. In this paper, we
provide the first systematic finite-sample study of proper scoring rules for
time-series forecasting evaluation. Through a power analysis, we identify the
"region of reliability" of a scoring rule, i.e., the set of practical
conditions where it can be relied on to identify forecasting errors. We carry
out our analysis on a comprehensive synthetic benchmark, specifically designed
to test several key discrepancies between ground-truth and forecast
distributions, and we gauge the generalizability of our findings to real-world
tasks with an application to an electricity production problem. Our results
reveal critical shortcomings in the evaluation of multivariate probabilistic
forecasts as commonly performed in the literature.
