  This paper will describe and analyze a new phenomenon that was not known
before, which we call "Early Transferability". Its essence is that the
adversarial perturbations transfer among different networks even at extremely
early stages in their training. In fact, one can initialize two networks with
two different independent choices of random weights and measure the angle
between their adversarial perturbations after each step of the training. What
we discovered was that these two adversarial directions started to align with
each other already after the first few training steps (which typically use only
a small fraction of the available training data), even though the accuracy of
the two networks hadn't started to improve from their initial bad values due to
the early stage of the training. The purpose of this paper is to present this
phenomenon experimentally and propose plausible explanations for some of its
properties.
