  In this paper, we propose a novel transfer-based targeted attack method that
optimizes the adversarial perturbations without any extra training efforts for
auxiliary networks on training data. Our new attack method is proposed based on
the observation that highly universal adversarial perturbations tend to be more
transferable for targeted attacks. Therefore, we propose to make the
perturbation to be agnostic to different local regions within one image, which
we called as self-universality. Instead of optimizing the perturbations on
different images, optimizing on different regions to achieve self-universality
can get rid of using extra data. Specifically, we introduce a feature
similarity loss that encourages the learned perturbations to be universal by
maximizing the feature similarity between adversarial perturbed global images
and randomly cropped local regions. With the feature similarity loss, our
method makes the features from adversarial perturbations to be more dominant
than that of benign images, hence improving targeted transferability. We name
the proposed attack method as Self-Universality (SU) attack. Extensive
experiments demonstrate that SU can achieve high success rates for
transfer-based targeted attacks. On ImageNet-compatible dataset, SU yields an
improvement of 12\% compared with existing state-of-the-art methods. Code is
available at https://github.com/zhipeng-wei/Self-Universality.
