  Industrial systems are increasingly threatened by cyberattacks with
potentially disastrous consequences. To counter such attacks, industrial
intrusion detection systems strive to timely uncover even the most
sophisticated breaches. Due to its criticality for society, this fast-growing
field attracts researchers from diverse backgrounds, resulting in 130 new
detection approaches in 2021 alone. This huge momentum facilitates the
exploration of diverse promising paths but likewise risks fragmenting the
research landscape and burying promising progress. Consequently, it needs sound
and comprehensible evaluations to mitigate this risk and catalyze efforts into
sustainable scientific progress with real-world applicability. In this paper,
we therefore systematically analyze the evaluation methodologies of this field
to understand the current state of industrial intrusion detection research. Our
analysis of 609 publications shows that the rapid growth of this research field
has positive and negative consequences. While we observe an increased use of
public datasets, publications still only evaluate 1.3 datasets on average, and
frequently used benchmarking metrics are ambiguous. At the same time, the
adoption of newly developed benchmarking metrics sees little advancement.
Finally, our systematic analysis enables us to provide actionable
recommendations for all actors involved and thus bring the entire research
field forward.
