  In this paper, we present a scientific evaluation of four prominent malware
detection tools to assist an organization with two primary questions: To what
extent do ML-based tools accurately classify previously- and never-before-seen
files? Is it worth purchasing a network-level malware detector? To identify
weaknesses, we tested each tool against 3,536 total files (2,554 or 72\%
malicious, 982 or 28\% benign) of a variety of file types, including hundreds
of malicious zero-days, polyglots, and APT-style files, delivered on multiple
protocols. We present statistical results on detection time and accuracy,
consider complementary analysis (using multiple tools together), and provide
two novel applications of the recent cost-benefit evaluation procedure of
Iannacone \& Bridges. While the ML-based tools are more effective at detecting
zero-day files and executables, the signature-based tool may still be an
overall better option. Both network-based tools provide substantial (simulated)
savings when paired with either host tool, yet both show poor detection rates
on protocols other than HTTP or SMTP. Our results show that all four tools have
near-perfect precision but alarmingly low recall, especially on file types
other than executables and office files -- 37% of malware tested, including all
polyglot files, were undetected. Priorities for researchers and takeaways for
end users are given.
