  The black-box adversarial attack has attracted impressive attention for its
practical use in the field of deep learning security. Meanwhile, it is very
challenging as there is no access to the network architecture or internal
weights of the target model. Based on the hypothesis that if an example remains
adversarial for multiple models, then it is more likely to transfer the attack
capability to other models, the ensemble-based adversarial attack methods are
efficient and widely used for black-box attacks. However, ways of ensemble
attack are rather less investigated, and existing ensemble attacks simply fuse
the outputs of all the models evenly. In this work, we treat the iterative
ensemble attack as a stochastic gradient descent optimization process, in which
the variance of the gradients on different models may lead to poor local
optima. To this end, we propose a novel attack method called the stochastic
variance reduced ensemble (SVRE) attack, which could reduce the gradient
variance of the ensemble models and take full advantage of the ensemble attack.
Empirical results on the standard ImageNet dataset demonstrate that the
proposed method could boost the adversarial transferability and outperforms
existing ensemble attacks significantly. Code is available at
https://github.com/JHL-HUST/SVRE.
