  The extreme fragility of deep neural networks, when presented with tiny
perturbations in their inputs, was independently discovered by several research
groups in 2013. However, despite enormous effort, these adversarial examples
remained a counterintuitive phenomenon with no simple testable explanation. In
this paper, we introduce a new conceptual framework for how the decision
boundary between classes evolves during training, which we call the {\em
Dimpled Manifold Model}. In particular, we demonstrate that training is divided
into two distinct phases. The first phase is a (typically fast) clinging
process in which the initially randomly oriented decision boundary gets very
close to the low dimensional image manifold, which contains all the training
examples. Next, there is a (typically slow) dimpling phase which creates
shallow bulges in the decision boundary that move it to the correct side of the
training examples. This framework provides a simple explanation for why
adversarial examples exist, why their perturbations have such tiny norms, and
why they look like random noise rather than like the target class. This
explanation is also used to show that a network that was adversarially trained
with incorrectly labeled images might still correctly classify most test
images, and to show that the main effect of adversarial training is just to
deepen the generated dimples in the decision boundary. Finally, we discuss and
demonstrate the very different properties of on-manifold and off-manifold
adversarial perturbations. We describe the results of numerous experiments
which strongly support this new model, using both low dimensional synthetic
datasets and high dimensional natural datasets.
