  This paper addresses the challenging black-box adversarial attack problem,
where only classification confidence of a victim model is available. Inspired
by consistency of visual saliency between different vision models, a surrogate
model is expected to improve the attack performance via transferability. By
combining transferability-based and query-based black-box attack, we propose a
surprisingly simple baseline approach (named SimBA++) using the surrogate
model, which significantly outperforms several state-of-the-art methods.
Moreover, to efficiently utilize the query feedback, we update the surrogate
model in a novel learning scheme, named High-Order Gradient Approximation
(HOGA). By constructing a high-order gradient computation graph, we update the
surrogate model to approximate the victim model in both forward and backward
pass. The SimBA++ and HOGA result in Learnable Black-Box Attack (LeBA), which
surpasses previous state of the art by considerable margins: the proposed LeBA
significantly reduces queries, while keeping higher attack success rates close
to 100% in extensive ImageNet experiments, including attacking vision
benchmarks and defensive models. Code is open source at
https://github.com/TrustworthyDL/LeBA.
