  As the world moves towards being increasingly dependent on computers and
automation, building secure applications, systems and networks are some of the
main challenges faced in the current decade. The number of threats that
individuals and businesses face is rising exponentially due to the increasing
complexity of networks and services of modern networks. To alleviate the impact
of these threats, researchers have proposed numerous solutions for anomaly
detection; however, current tools often fail to adapt to ever-changing
architectures, associated threats and zero-day attacks. This manuscript aims to
pinpoint research gaps and shortcomings of current datasets, their impact on
building Network Intrusion Detection Systems (NIDS) and the growing number of
sophisticated threats. To this end, this manuscript provides researchers with
two key pieces of information; a survey of prominent datasets, analyzing their
use and impact on the development of the past decade's Intrusion Detection
Systems (IDS) and a taxonomy of network threats and associated tools to carry
out these attacks. The manuscript highlights that current IDS research covers
only 33.3% of our threat taxonomy. Current datasets demonstrate a clear lack of
real-network threats, attack representation and include a large number of
deprecated threats, which together limit the detection accuracy of current
machine learning IDS approaches. The unique combination of the taxonomy and the
analysis of the datasets provided in this manuscript aims to improve the
creation of datasets and the collection of real-world data. As a result, this
will improve the efficiency of the next generation IDS and reflect network
threats more accurately within new datasets.
