  Driven by the proliferation of real-world application scenarios and scales,
time series anomaly detection (TSAD) has attracted considerable scholarly and
industrial interest. However, existing algorithms exhibit a gap in terms of
training paradigm, online detection paradigm, and evaluation criteria when
compared to the actual needs of real-world industrial systems. Firstly, current
algorithms typically train a specific model for each individual time series. In
a large-scale online system with tens of thousands of curves, maintaining such
a multitude of models is impractical. The performance of using merely one
single unified model to detect anomalies remains unknown. Secondly, most TSAD
models are trained on the historical part of a time series and are tested on
its future segment. In distributed systems, however, there are frequent system
deployments and upgrades, with new, previously unseen time series emerging
daily. The performance of testing newly incoming unseen time series on current
TSAD algorithms remains unknown. Lastly, although some papers have conducted
detailed surveys, the absence of an online evaluation platform prevents
answering questions like "Who is the best at anomaly detection at the current
stage?" In this paper, we propose TimeSeriesBench, an industrial-grade
benchmark that we continuously maintain as a leaderboard. On this leaderboard,
we assess the performance of existing algorithms across more than 168
evaluation settings combining different training and testing paradigms,
evaluation metrics and datasets. Through our comprehensive analysis of the
results, we provide recommendations for the future design of anomaly detection
algorithms. To address known issues with existing public datasets, we release
an industrial dataset to the public together with TimeSeriesBench. All code,
data, and the online leaderboard have been made publicly available.
