  Deep neural networks (DNNs) have been shown to tolerate "brain damage":
cumulative changes to the network's parameters (e.g., pruning, numerical
perturbations) typically result in a graceful degradation of classification
accuracy. However, the limits of this natural resilience are not well
understood in the presence of small adversarial changes to the DNN parameters'
underlying memory representation, such as bit-flips that may be induced by
hardware fault attacks. We study the effects of bitwise corruptions on 19 DNN
models---six architectures on three image classification tasks---and we show
that most models have at least one parameter that, after a specific bit-flip in
their bitwise representation, causes an accuracy loss of over 90%. We employ
simple heuristics to efficiently identify the parameters likely to be
vulnerable. We estimate that 40-50% of the parameters in a model might lead to
an accuracy drop greater than 10% when individually subjected to such
single-bit perturbations. To demonstrate how an adversary could take advantage
of this vulnerability, we study the impact of an exemplary hardware fault
attack, Rowhammer, on DNNs. Specifically, we show that a Rowhammer enabled
attacker co-located in the same physical machine can inflict significant
accuracy drops (up to 99%) even with single bit-flip corruptions and no
knowledge of the model. Our results expose the limits of DNNs' resilience
against parameter perturbations induced by real-world fault attacks. We
conclude by discussing possible mitigations and future research directions
towards fault attack-resilient DNNs.
