  Deep neural networks (DNNs) are at the forefront of cutting-edge technology,
and have been achieving remarkable performance in a variety of complex tasks.
Nevertheless, their integration into safety-critical systems, such as in the
aerospace or automotive domains, poses a significant challenge due to the
threat of adversarial inputs: perturbations in inputs that might cause the DNN
to make grievous mistakes. Multiple studies have demonstrated that even modern
DNNs are susceptible to adversarial inputs, and this risk must thus be measured
and mitigated to allow the deployment of DNNs in critical settings. Here, we
present gRoMA (global Robustness Measurement and Assessment), an innovative and
scalable tool that implements a probabilistic approach to measure the global
categorial robustness of a DNN. Specifically, gRoMA measures the probability of
encountering adversarial inputs for a specific output category. Our tool
operates on pre-trained, black-box classification DNNs, and generates input
samples belonging to an output category of interest. It measures the DNN's
susceptibility to adversarial inputs around these inputs, and aggregates the
results to infer the overall global categorial robustness of the DNN up to some
small bounded statistical error.
  We evaluate our tool on the popular Densenet DNN model over the CIFAR10
dataset. Our results reveal significant gaps in the robustness of the different
output categories. This experiment demonstrates the usefulness and scalability
of our approach and its potential for allowing DNNs to be deployed within
critical systems of interest.
