  Anomaly-based intrusion detection promises to detect novel or unknown attacks
on industrial control systems by modeling expected system behavior and raising
corresponding alarms for any deviations.As manually creating these behavioral
models is tedious and error-prone, research focuses on machine learning to
train them automatically, achieving detection rates upwards of 99%. However,
these approaches are typically trained not only on benign traffic but also on
attacks and then evaluated against the same type of attack used for training.
Hence, their actual, real-world performance on unknown (not trained on) attacks
remains unclear. In turn, the reported near-perfect detection rates of machine
learning-based intrusion detection might create a false sense of security. To
assess this situation and clarify the real potential of machine learning-based
industrial intrusion detection, we develop an evaluation methodology and
examine multiple approaches from literature for their performance on unknown
attacks (excluded from training). Our results highlight an ineffectiveness in
detecting unknown attacks, with detection rates dropping to between 3.2% and
14.7% for some types of attacks. Moving forward, we derive recommendations for
further research on machine learning-based approaches to ensure clarity on
their ability to detect unknown attacks.
