  Analysis techniques are critical for gaining insight into network traffic
given both the higher proportion of encrypted traffic and increasing data
rates. Unfortunately, the domain of network traffic analysis suffers from a
lack of standardization, leading to incomparable results and barriers to
reproducibility. Unlike other disciplines, no standard dataset format exists,
forcing researchers and practitioners to create bespoke analysis pipelines for
each individual task. Without standardization researchers cannot compare
"apples-to-apples", preventing us from knowing with certainty if a new
technique represents a methodological advancement or if it simply benefits from
a different interpretation of a given dataset.
  In this work, we examine irreproducibility that arises from the lack of
standardization in network traffic analysis. First, we study the literature,
highlighting evidence of irreproducible research based on different
interpretations of popular public datasets. Next, we investigate the underlying
issues that have lead to the status quo and prevent reproducible research.
Third, we outline the standardization requirements that any solution aiming to
fix reproducibility issues must address. We then introduce pcapML, an open
source system which increases reproducibility of network traffic analysis
research by enabling metadata information to be directly encoded into raw
traffic captures in a generic manner. Finally, we use the standardization
pcapML provides to create the pcapML benchmarks, an open source leaderboard
website and repository built to track the progress of network traffic analysis
methods.
