  The transferability of adversarial perturbations provides an effective
shortcut for black-box attacks. Targeted perturbations have greater
practicality but are more difficult to transfer between models. In this paper,
we experimentally and theoretically demonstrated that neural networks trained
on the same dataset have more consistent performance in
High-Sample-Density-Regions (HSDR) of each class instead of low sample density
regions. Therefore, in the target setting, adding perturbations towards HSDR of
the target class is more effective in improving transferability. However,
density estimation is challenging in high-dimensional scenarios. Further
theoretical and experimental verification demonstrates that easy samples with
low loss are more likely to be located in HSDR. Perturbations towards such easy
samples in the target class can avoid density estimation for HSDR location.
Based on the above facts, we verified that adding perturbations to easy samples
in the target class improves targeted adversarial transferability of existing
attack methods. A generative targeted attack strategy named Easy Sample
Matching Attack (ESMA) is proposed, which has a higher success rate for
targeted attacks and outperforms the SOTA generative method. Moreover, ESMA
requires only 5% of the storage space and much less computation time comparing
to the current SOTA, as ESMA attacks all classes with only one model instead of
seperate models for each class. Our code is available at
https://github.com/gjq100/ESMA.
