  We introduce the Robustness of Hierarchically Organized Time Series (RHiOTS)
framework, designed to assess the robustness of hierarchical time series
forecasting models and algorithms on real-world datasets. Hierarchical time
series, where lower-level forecasts must sum to upper-level ones, are prevalent
in various contexts, such as retail sales across countries. Current empirical
evaluations of forecasting methods are often limited to a small set of
benchmark datasets, offering a narrow view of algorithm behavior. RHiOTS
addresses this gap by systematically altering existing datasets and modifying
the characteristics of individual series and their interrelations. It uses a
set of parameterizable transformations to simulate those changes in the data
distribution. Additionally, RHiOTS incorporates an innovative visualization
component, turning complex, multidimensional robustness evaluation results into
intuitive, easily interpretable visuals. This approach allows an in-depth
analysis of algorithm and model behavior under diverse conditions. We
illustrate the use of RHiOTS by analyzing the predictive performance of several
algorithms. Our findings show that traditional statistical methods are more
robust than state-of-the-art deep learning algorithms, except when the
transformation effect is highly disruptive. Furthermore, we found no
significant differences in the robustness of the algorithms when applying
specific reconciliation methods, such as MinT. RHiOTS provides researchers with
a comprehensive tool for understanding the nuanced behavior of forecasting
algorithms, offering a more reliable basis for selecting the most appropriate
method for a given problem.
