  Deep neural networks have shown their vulnerability to adversarial attacks.
In this paper, we focus on sparse adversarial attack based on the $\ell_0$ norm
constraint, which can succeed by only modifying a few pixels of an image.
Despite a high attack success rate, prior sparse attack methods achieve a low
transferability under the black-box protocol due to overfitting the target
model. Therefore, we introduce a generator architecture to alleviate the
overfitting issue and thus efficiently craft transferable sparse adversarial
examples. Specifically, the generator decouples the sparse perturbation into
amplitude and position components. We carefully design a random quantization
operator to optimize these two components jointly in an end-to-end way. The
experiment shows that our method has improved the transferability by a large
margin under a similar sparsity setting compared with state-of-the-art methods.
Moreover, our method achieves superior inference speed, 700$\times$ faster than
other optimization-based methods. The code is available at
https://github.com/shaguopohuaizhe/TSAA.
