  Adversarial attacks perturb images such that a deep neural network produces
incorrect classification results. A promising approach to defend against
adversarial attacks on natural multi-object scenes is to impose a
context-consistency check, wherein, if the detected objects are not consistent
with an appropriately defined context, then an attack is suspected. Stronger
attacks are needed to fool such context-aware detectors. We present the first
approach for generating context-consistent adversarial attacks that can evade
the context-consistency check of black-box object detectors operating on
complex, natural scenes. Unlike many black-box attacks that perform repeated
attempts and open themselves to detection, we assume a "zero-query" setting,
where the attacker has no knowledge of the classification decisions of the
victim system. First, we derive multiple attack plans that assign incorrect
labels to victim objects in a context-consistent manner. Then we design and use
a novel data structure that we call the perturbation success probability
matrix, which enables us to filter the attack plans and choose the one most
likely to succeed. This final attack plan is implemented using a
perturbation-bounded adversarial attack algorithm. We compare our zero-query
attack against a few-query scheme that repeatedly checks if the victim system
is fooled. We also compare against state-of-the-art context-agnostic attacks.
Against a context-aware defense, the fooling rate of our zero-query approach is
significantly higher than context-agnostic approaches and higher than that
achievable with up to three rounds of the few-query scheme.
