  Detection of anomalous situations for complex mission-critical systems holds
paramount importance when their service continuity needs to be ensured. A major
challenge in detecting anomalies from the operational data arises due to the
imbalanced class distribution problem since the anomalies are supposed to be
rare events. This paper evaluates a diverse array of machine learning-based
anomaly detection algorithms through a comprehensive benchmark study. The paper
contributes significantly by conducting an unbiased comparison of various
anomaly detection algorithms, spanning classical machine learning including
various tree-based approaches to deep learning and outlier detection methods.
The inclusion of 104 publicly available and a few proprietary industrial
systems datasets enhances the diversity of the study, allowing for a more
realistic evaluation of algorithm performance and emphasizing the importance of
adaptability to real-world scenarios. The paper dispels the deep learning myth,
demonstrating that though powerful, deep learning is not a universal solution
in this case. We observed that recently proposed tree-based evolutionary
algorithms outperform in many scenarios. We noticed that tree-based approaches
catch a singleton anomaly in a dataset where deep learning methods fail. On the
other hand, classical SVM performs the best on datasets with more than 10%
anomalies, implying that such scenarios can be best modeled as a classification
problem rather than anomaly detection. To our knowledge, such a study on a
large number of state-of-the-art algorithms using diverse data sets, with the
objective of guiding researchers and practitioners in making informed
algorithmic choices, has not been attempted earlier.
