  Most research in the area of intrusion detection requires datasets to
develop, evaluate or compare systems in one way or another. In this field,
however, finding suitable datasets is a challenge on to itself. Most publicly
available datasets have negative qualities that limit their usefulness. In this
article, we propose ID2T (Intrusion Detection Dataset Toolkit) to tackle this
problem. ID2T facilitates the creation of labeled datasets by injecting
synthetic attacks into background traffic. The injected synthetic attacks blend
themselves with the background traffic by mimicking the background traffic's
properties to eliminate any trace of ID2T's usage.
  This work has three core contribution areas. First, we present a
comprehensive survey on intrusion detection datasets. In the survey, we propose
a classification to group the negative qualities we found in the datasets.
Second, the architecture of ID2T is revised, improved and expanded. The
architectural changes enable ID2T to inject recent and advanced attacks such as
the widespread EternalBlue exploit or botnet communication patterns. The
toolkit's new functionality provides a set of tests, known as TIDED (Testing
Intrusion Detection Datasets), that help identify potential defects in the
background traffic into which attacks are injected. Third, we illustrate how
ID2T is used in different use-case scenarios to evaluate the performance of
anomaly and signature-based intrusion detection systems in a reproducible
manner. ID2T is open source software and is made available to the community to
expand its arsenal of attacks and capabilities.
