  Time series are generated in diverse domains such as economic, traffic,
health, and energy, where forecasting of future values has numerous important
applications. Not surprisingly, many forecasting methods are being proposed. To
ensure progress, it is essential to be able to study and compare such methods
empirically in a comprehensive and reliable manner. To achieve this, we propose
TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB
advances the state-of-the-art by addressing shortcomings related to datasets,
comparison methods, and evaluation pipelines: 1) insufficient coverage of data
domains, 2) stereotype bias against traditional methods, and 3) inconsistent
and inflexible pipelines. To achieve better domain coverage, we include
datasets from 10 different domains: traffic, electricity, energy, the
environment, nature, economic, stock markets, banking, health, and the web. We
also provide a time series characterization to ensure that the selected
datasets are comprehensive. To remove biases against some methods, we include a
diverse range of methods, including statistical learning, machine learning, and
deep learning methods, and we also support a variety of evaluation strategies
and metrics to ensure a more comprehensive evaluations of different methods. To
support the integration of different methods into the benchmark and enable fair
comparisons, TFB features a flexible and scalable pipeline that eliminates
biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate
Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14
Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The
benchmark code and data are available at
https://github.com/decisionintelligence/TFB.
