  Testing with randomly generated inputs (fuzzing) has gained significant
traction due to its capacity to expose program vulnerabilities automatically.
Fuzz testing campaigns generate large amounts of data, making them ideal for
the application of machine learning (ML). Neural program smoothing (NPS), a
specific family of ML-guided fuzzers, aims to use a neural network as a smooth
approximation of the program target for new test case generation.
  In this paper, we conduct the most extensive evaluation of NPS fuzzers
against standard gray-box fuzzers (>11 CPU years and >5.5 GPU years), and make
the following contributions: (1) We find that the original performance claims
for NPS fuzzers do not hold; a gap we relate to fundamental, implementation,
and experimental limitations of prior works. (2) We contribute the first
in-depth analysis of the contribution of machine learning and gradient-based
mutations in NPS. (3) We implement Neuzz++, which shows that addressing the
practical limitations of NPS fuzzers improves performance, but that standard
gray-box fuzzers almost always surpass NPS-based fuzzers. (4) As a consequence,
we propose new guidelines targeted at benchmarking fuzzing based on machine
learning, and present MLFuzz, a platform with GPU access for easy and
reproducible evaluation of ML-based fuzzers. Neuzz++, MLFuzz, and all our data
are public.
