  We propose an approach for adversarial attacks on dense prediction models
(such as object detectors and segmentation). It is well known that the attacks
generated by a single surrogate model do not transfer to arbitrary (blackbox)
victim models. Furthermore, targeted attacks are often more challenging than
the untargeted attacks. In this paper, we show that a carefully designed
ensemble can create effective attacks for a number of victim models. In
particular, we show that normalization of the weights for individual models
plays a critical role in the success of the attacks. We then demonstrate that
by adjusting the weights of the ensemble according to the victim model can
further improve the performance of the attacks. We performed a number of
experiments for object detectors and segmentation to highlight the significance
of the our proposed methods. Our proposed ensemble-based method outperforms
existing blackbox attack methods for object detection and segmentation. Finally
we show that our proposed method can also generate a single perturbation that
can fool multiple blackbox detection and segmentation models simultaneously.
Code is available at https://github.com/CSIPlab/EBAD.
