  Intermediate-level attacks that attempt to perturb feature representations
following an adversarial direction drastically have shown favorable performance
in crafting transferable adversarial examples. Existing methods in this
category are normally formulated with two separate stages, where a directional
guide is required to be determined at first and the scalar projection of the
intermediate-level perturbation onto the directional guide is enlarged
thereafter. The obtained perturbation deviates from the guide inevitably in the
feature space, and it is revealed in this paper that such a deviation may lead
to sub-optimal attack. To address this issue, we develop a novel
intermediate-level method that crafts adversarial examples within a single
stage of optimization. In particular, the proposed method, named
intermediate-level perturbation decay (ILPD), encourages the intermediate-level
perturbation to be in an effective adversarial direction and to possess a great
magnitude simultaneously. In-depth discussion verifies the effectiveness of our
method. Experimental results show that it outperforms state-of-the-arts by
large margins in attacking various victim models on ImageNet (+10.07% on
average) and CIFAR-10 (+3.88% on average). Our code is at
https://github.com/qizhangli/ILPD-attack.
