  Adversarial patch is an important form of real-world adversarial attack that
brings serious risks to the robustness of deep neural networks. Previous
methods generate adversarial patches by either optimizing their perturbation
values while fixing the pasting position or manipulating the position while
fixing the patch's content. This reveals that the positions and perturbations
are both important to the adversarial attack. For that, in this paper, we
propose a novel method to simultaneously optimize the position and perturbation
for an adversarial patch, and thus obtain a high attack success rate in the
black-box setting. Technically, we regard the patch's position, the
pre-designed hyper-parameters to determine the patch's perturbations as the
variables, and utilize the reinforcement learning framework to simultaneously
solve for the optimal solution based on the rewards obtained from the target
model with a small number of queries. Extensive experiments are conducted on
the Face Recognition (FR) task, and results on four representative FR models
show that our method can significantly improve the attack success rate and
query efficiency. Besides, experiments on the commercial FR service and
physical environments confirm its practical application value. We also extend
our method to the traffic sign recognition task to verify its generalization
ability.
