  This paper investigates an adversary's ease of attack in generating
adversarial examples for real-world scenarios. We address three key
requirements for practical attacks for the real-world: 1) automatically
constraining the size and shape of the attack so it can be applied with
stickers, 2) transform-robustness, i.e., robustness of a attack to
environmental physical variations such as viewpoint and lighting changes, and
3) supporting attacks in not only white-box, but also black-box hard-label
scenarios, so that the adversary can attack proprietary models. In this work,
we propose GRAPHITE, an efficient and general framework for generating attacks
that satisfy the above three key requirements. GRAPHITE takes advantage of
transform-robustness, a metric based on expectation over transforms (EoT), to
automatically generate small masks and optimize with gradient-free
optimization. GRAPHITE is also flexible as it can easily trade-off
transform-robustness, perturbation size, and query count in black-box settings.
On a GTSRB model in a hard-label black-box setting, we are able to find attacks
on all possible 1,806 victim-target class pairs with averages of 77.8%
transform-robustness, perturbation size of 16.63% of the victim images, and
126K queries per pair. For digital-only attacks where achieving
transform-robustness is not a requirement, GRAPHITE is able to find successful
small-patch attacks with an average of only 566 queries for 92.2% of
victim-target pairs. GRAPHITE is also able to find successful attacks using
perturbations that modify small areas of the input image against PatchGuard, a
recently proposed defense against patch-based attacks.
