  Transfer adversarial attack is a non-trivial black-box adversarial attack
that aims to craft adversarial perturbations on the surrogate model and then
apply such perturbations to the victim model. However, the transferability of
perturbations from existing methods is still limited, since the adversarial
perturbations are easily overfitting with a single surrogate model and specific
data pattern. In this paper, we propose a Learning to Learn Transferable Attack
(LLTA) method, which makes the adversarial perturbations more generalized via
learning from both data and model augmentation. For data augmentation, we adopt
simple random resizing and padding. For model augmentation, we randomly alter
the back propagation instead of the forward propagation to eliminate the effect
on the model prediction. By treating the attack of both specific data and a
modified model as a task, we expect the adversarial perturbations to adopt
enough tasks for generalization. To this end, the meta-learning algorithm is
further introduced during the iteration of perturbation generation. Empirical
results on the widely-used dataset demonstrate the effectiveness of our attack
method with a 12.85% higher success rate of transfer attack compared with the
state-of-the-art methods. We also evaluate our method on the real-world online
system, i.e., Google Cloud Vision API, to further show the practical potentials
of our method.
