  Deep Learning Systems (DLSs) have been widely applied in safety-critical
tasks such as autopilot. However, when a perturbed input is fed into a DLS for
inference, the DLS often has incorrect outputs (i.e., faults). DLS testing
techniques (e.g., DeepXplore) detect such faults by generating perturbed inputs
to explore data flows that induce faults. Since a DLS often has infinitely many
data flows, existing techniques require developers to manually specify a set of
activation values in a DLS's neurons for exploring fault-inducing data flows.
Unfortunately, recent studies show that such manual effort is tedious and can
detect only a tiny proportion of fault-inducing data flows.
  In this paper, we present Themis, the first automatic DLS testing system,
which attains strong fault detection capability by ensuring a full coverage of
fault-inducing data flows at a high probability. Themis carries a new workflow
for automatically and systematically revealing data flows whose internal
neurons' outputs vary substantially when the inputs are slightly perturbed, as
these data flows are likely fault-inducing. We evaluated Themis on ten
different DLSs and found that on average the number of faults detected by
Themis was 3.78X more than four notable DLS testing techniques. By retraining
all evaluated DLSs with the detected faults, Themis also increased (regained)
these DLSs' accuracies on average 14.7X higher than all baselines.
