  Blackbox transfer attacks for image classifiers have been extensively studied
in recent years. In contrast, little progress has been made on transfer attacks
for object detectors. Object detectors take a holistic view of the image and
the detection of one object (or lack thereof) often depends on other objects in
the scene. This makes such detectors inherently context-aware and adversarial
attacks in this space are more challenging than those targeting image
classifiers. In this paper, we present a new approach to generate context-aware
attacks for object detectors. We show that by using co-occurrence of objects
and their relative locations and sizes as context information, we can
successfully generate targeted mis-categorization attacks that achieve higher
transfer success rates on blackbox object detectors than the state-of-the-art.
We test our approach on a variety of object detectors with images from PASCAL
VOC and MS COCO datasets and demonstrate up to $20$ percentage points
improvement in performance compared to the other state-of-the-art methods.
