  Despite the empirical success of using Adversarial Training to defend deep
learning models against adversarial perturbations, so far, it still remains
rather unclear what the principles are behind the existence of adversarial
perturbations, and what adversarial training does to the neural network to
remove them.
  In this paper, we present a principle that we call Feature Purification,
where we show one of the causes of the existence of adversarial examples is the
accumulation of certain small dense mixtures in the hidden weights during the
training process of a neural network; and more importantly, one of the goals of
adversarial training is to remove such mixtures to purify hidden weights. We
present both experiments on the CIFAR-10 dataset to illustrate this principle,
and a theoretical result proving that for certain natural classification tasks,
training a two-layer neural network with ReLU activation using randomly
initialized gradient descent indeed satisfies this principle.
  Technically, we give, to the best of our knowledge, the first result proving
that the following two can hold simultaneously for training a neural network
with ReLU activation. (1) Training over the original data is indeed non-robust
to small adversarial perturbations of some radius. (2) Adversarial training,
even with an empirical perturbation algorithm such as FGM, can in fact be
provably robust against ANY perturbations of the same radius. Finally, we also
prove a complexity lower bound, showing that low complexity models such as
linear classifiers, low-degree polynomials, or even the neural tangent kernel
for this network, CANNOT defend against perturbations of this same radius, no
matter what algorithms are used to train them.
