  Detecting anomalous behavior in network traffic is a major challenge due to
the volume and velocity of network traffic. For example, a 10 Gigabit Ethernet
connection can generate over 50 MB/s of packet headers. For global network
providers, this challenge can be amplified by many orders of magnitude.
Development of novel computer network traffic analytics requires: high level
programming environments, massive amount of packet capture (PCAP) data, and
diverse data products for "at scale" algorithm pipeline development. D4M
(Dynamic Distributed Dimensional Data Model) combines the power of sparse
linear algebra, associative arrays, parallel processing, and distributed
databases (such as SciDB and Apache Accumulo) to provide a scalable data and
computation system that addresses the big data problems associated with network
analytics development. Combining D4M with the MIT SuperCloud manycore
processors and parallel storage system enables network analysts to
interactively process massive amounts of data in minutes. To demonstrate these
capabilities, we have implemented a representative analytics pipeline in D4M
and benchmarked it on 96 hours of Gigabit PCAP data with MIT SuperCloud. The
entire pipeline from uncompressing the raw files to database ingest was
implemented in 135 lines of D4M code and achieved speedups of over 20,000.
