  Transfer-based targeted adversarial attacks against black-box deep neural
networks (DNNs) have been proven to be significantly more challenging than
untargeted ones. The impressive transferability of current SOTA, the generative
methods, comes at the cost of requiring massive amounts of additional data and
time-consuming training for each targeted label. This results in limited
efficiency and flexibility, significantly hindering their deployment in
practical applications. In this paper, we offer a self-universal perspective
that unveils the great yet underexplored potential of input transformations in
pursuing this goal. Specifically, transformations universalize gradient-based
attacks with intrinsic but overlooked semantics inherent within individual
images, exhibiting similar scalability and comparable results to time-consuming
learning over massive additional data from diverse classes. We also contribute
a surprising empirical insight that one of the most fundamental
transformations, simple image scaling, is highly effective, scalable,
sufficient, and necessary in enhancing targeted transferability. We further
augment simple scaling with orthogonal transformations and block-wise
applicability, resulting in the Simple, faSt, Self-universal yet Strong Scale
Transformation (S$^4$ST) for self-universal TTA. On the ImageNet-Compatible
benchmark dataset, our method achieves a 19.8% improvement in the average
targeted transfer success rate against various challenging victim models over
existing SOTA transformation methods while only consuming 36% time for
attacking. It also outperforms resource-intensive attacks by a large margin in
various challenging settings.
