  We consider the hard label based black box adversarial attack setting which
solely observes predicted classes from the target model. Most of the attack
methods in this setting suffer from impractical number of queries required to
achieve a successful attack. One approach to tackle this drawback is utilising
the adversarial transferability between white box surrogate models and black
box target model. However, the majority of the methods adopting this approach
are soft label based to take the full advantage of zeroth order optimisation.
Unlike mainstream methods, we propose a new practical setting of hard label
based attack with an optimisation process guided by a pretrained surrogate
model. Experiments show the proposed method significantly improves the query
efficiency of the hard label based black-box attack across various target model
architectures. We find the proposed method achieves approximately 5 times
higher attack success rate compared to the benchmarks, especially at the small
query budgets as 100 and 250.
