  Accurate evaluation of forecasting models is essential for ensuring reliable
predictions. Current practices for evaluating and comparing forecasting models
focus on summarising performance into a single score, using metrics such as
SMAPE. We hypothesize that averaging performance over all samples dilutes
relevant information about the relative performance of models. Particularly,
conditions in which this relative performance is different than the overall
accuracy. We address this limitation by proposing a novel framework for
evaluating univariate time series forecasting models from multiple
perspectives, such as one-step ahead forecasting versus multi-step ahead
forecasting. We show the advantages of this framework by comparing a
state-of-the-art deep learning approach with classical forecasting techniques.
While classical methods (e.g. ARIMA) are long-standing approaches to
forecasting, deep neural networks (e.g. NHITS) have recently shown
state-of-the-art forecasting performance in benchmark datasets. We conducted
extensive experiments that show NHITS generally performs best, but its
superiority varies with forecasting conditions. For instance, concerning the
forecasting horizon, NHITS only outperforms classical approaches for multi-step
ahead forecasting. Another relevant insight is that, when dealing with
anomalies, NHITS is outperformed by methods such as Theta. These findings
highlight the importance of aspect-based model evaluation.
