  COVID-19 has been a public health emergency of international concern since
early 2020. Reliable forecasting is critical to diminish the impact of this
disease. To date, a large number of different forecasting models have been
proposed, mainly including statistical models, compartmental models, and deep
learning models. However, due to various uncertain factors across different
regions such as economics and government policy, no forecasting model appears
to be the best for all scenarios. In this paper, we perform quantitative
analysis of COVID-19 forecasting of confirmed cases and deaths across different
regions in the United States with different forecasting horizons, and evaluate
the relative impacts of the following three dimensions on the predictive
performance (improvement and variation) through different evaluation metrics:
model selection, hyperparameter tuning, and the length of time series required
for training. We find that if a dimension brings about higher performance
gains, if not well-tuned, it may also lead to harsher performance penalties.
Furthermore, model selection is the dominant factor in determining the
predictive performance. It is responsible for both the largest improvement and
the largest variation in performance in all prediction tasks across different
regions. While practitioners may perform more complicated time series analysis
in practice, they should be able to achieve reasonable results if they have
adequate insight into key decisions like model selection.
