  Adversarial examples represent a serious threat for deep neural networks in
several application domains and a huge amount of work has been produced to
investigate them and mitigate their effects. Nevertheless, no much work has
been devoted to the generation of datasets specifically designed to evaluate
the adversarial robustness of neural models. This paper presents CARLA-GeAR, a
tool for the automatic generation of photo-realistic synthetic datasets that
can be used for a systematic evaluation of the adversarial robustness of neural
models against physical adversarial patches, as well as for comparing the
performance of different adversarial defense/detection methods. The tool is
built on the CARLA simulator, using its Python API, and allows the generation
of datasets for several vision tasks in the context of autonomous driving. The
adversarial patches included in the generated datasets are attached to
billboards or the back of a truck and are crafted by using state-of-the-art
white-box attack strategies to maximize the prediction error of the model under
test. Finally, the paper presents an experimental study to evaluate the
performance of some defense methods against such attacks, showing how the
datasets generated with CARLA-GeAR might be used in future work as a benchmark
for adversarial defense in the real world. All the code and datasets used in
this paper are available at http://carlagear.retis.santannapisa.it.
