  Multivariate Time Series (MVTS) anomaly detection is a long-standing and
challenging research topic that has attracted tremendous research effort from
both industry and academia recently. However, a careful study of the literature
makes us realize that 1) the community is active but not as organized as other
sibling machine learning communities such as Computer Vision (CV) and Natural
Language Processing (NLP), and 2) most proposed solutions are evaluated using
either inappropriate or highly flawed protocols, with an apparent lack of
scientific foundation. So flawed is one very popular protocol, the so-called
point-adjust protocol, that a random guess can be shown to systematically
outperform all algorithms developed so far. In this paper, we review and
evaluate many recent algorithms using more robust protocols and discuss how a
normally good protocol may have weaknesses in the context of MVTS anomaly
detection and how to mitigate them. We also share our concerns about benchmark
datasets, experiment design and evaluation methodology we observe in many
works. Furthermore, we propose a simple, yet challenging, baseline based on
Principal Components Analysis (PCA) that surprisingly outperforms many recent
Deep Learning (DL) based approaches on popular benchmark datasets. The main
objective of this work is to stimulate more effort towards important aspects of
the research such as data, experiment design, evaluation methodology and result
interpretability, instead of putting the highest weight on the design of
increasingly more complex and "fancier" algorithms.
