  Background: Over the past few decades, numerous forecasting methods have been
proposed in the field of epidemic forecasting. Such methods can be classified
into different categories such as deterministic vs. probabilistic, comparative
methods vs. generative methods, and so on. In some of the more popular
comparative methods, researchers compare observed epidemiological data from
early stages of an outbreak with the output of proposed models to forecast the
future trend and prevalence of the pandemic. A significant problem in this area
is the lack of standard well-defined evaluation measures to select the best
algorithm among different ones, as well as for selecting the best possible
configuration for a particular algorithm.
  Results: In this paper, we present an evaluation framework which allows for
combining different features, error measures, and ranking schema to evaluate
forecasts. We describe the various epidemic features (Epi-features) included to
characterize the output of forecasting methods and provide suitable error
measures that could be used to evaluate the accuracy of the methods with
respect to these Epi-features. We focus on long-term predictions rather than
short-term forecasting and demonstrate the utility of the framework by
evaluating six forecasting methods for predicting influenza in the United
States. Our results demonstrate that different error measures lead to different
rankings even for a single Epi-feature. Further, our experimental analyses show
that no single method dominates the rest in predicting all Epi-features, when
evaluated across error measures. As an alternative, we provide various
consensus ranking schema that summarizes individual rankings, thus accounting
for different error measures. We believe that a comprehensive evaluation
framework, as presented in this paper, will add value to the computational
epidemiology community.
