  This paper examines the current state of the science underlying cybersecurity
research with an emphasis on the non-signature-based intrusion detection
domain. First, the paper re-examines the base-rate fallacy originally published
by Axelsson, putting the impact of false positives into context. Given the
relative high numbers of false positives, the paper argues for deeper analysis
of false positives, akin to the analysis that true positives are treated to.
The second section of the paper examines the metrics being used to analyze
non-signature intrusion detection techniques, the current status quo of
employed metrics, and the impact of the status quo on scientific advancement.
Finally, the paper analyzes the use of online attack graphs and their
applicability, especially in scenarios of constrained environments, such as
Internet of Things devices. The use of offline attack graphs in such
constrained environments is also examined. In essence, a deep dive review
identified multiple areas throughout the field in which the effectiveness and
validity of the scientific method can be greatly improved, e.g., through
removal of logical fallacies.
