  Research has shown that widely used deep neural networks are vulnerable to
carefully crafted adversarial perturbations. Moreover, these adversarial
perturbations often transfer across models. We hypothesize that adversarial
weakness is composed of three sources of bias: architecture, dataset, and
random initialization. We show that one can decompose adversarial examples into
an architecture-dependent component, data-dependent component, and
noise-dependent component and that these components behave intuitively. For
example, noise-dependent components transfer poorly to all other models, while
architecture-dependent components transfer better to retrained models with the
same architecture. In addition, we demonstrate that these components can be
recombined to improve transferability without sacrificing efficacy on the
original model.
