  This paper provides a formal evaluation of the predictive performance of a
model (and its various updates) developed by the Institute for Health Metrics
and Evaluation (IHME) for predicting daily deaths attributed to COVID19 for
each state in the United States. The IHME models have received extensive
attention in social and mass media, and have influenced policy makers at the
highest levels of the United States government. For effective policy making the
accurate assessment of uncertainty, as well as accurate point predictions, are
necessary because the risks inherent in a decision must be taken into account,
especially in the present setting of a novel disease affecting millions of
lives. To assess the accuracy of the IHME models, we examine both forecast
accuracy as well as the predictive performance of the 95% prediction intervals
provided by the IHME models. We find that the initial IHME model underestimates
the uncertainty surrounding the number of daily deaths substantially.
Specifically, the true number of next day deaths fell outside the IHME
prediction intervals as much as 70% of the time, in comparison to the expected
value of 5%. In addition, we note that the performance of the initial model
does not improve with shorter forecast horizons. Regarding the updated models,
our analyses indicate that the later models do not show any improvement in the
accuracy of the point estimate predictions. In fact, there is some evidence
that this accuracy has actually decreased over the initial models. Moreover,
when considering the updated models, while we observe a larger percentage of
states having actual values lying inside the 95% prediction intervals (PI), our
analysis suggests that this observation may be attributed to the widening of
the PIs. The width of these intervals calls into question the usefulness of the
predictions to drive policy making and resource allocation.
