  Studying the robustness of machine learning models is important to ensure
consistent model behaviour across real-world settings. To this end, adversarial
robustness is a standard framework, which views robustness of predictions
through a binary lens: either a worst-case adversarial misclassification exists
in the local region around an input, or it does not. However, this binary
perspective does not account for the degrees of vulnerability, as data points
with a larger number of misclassified examples in their neighborhoods are more
vulnerable. In this work, we consider a complementary framework for robustness,
called average-case robustness, which measures the fraction of points in a
local region that provides consistent predictions. However, computing this
quantity is hard, as standard Monte Carlo approaches are inefficient especially
for high-dimensional inputs. In this work, we propose the first analytical
estimators for average-case robustness for multi-class classifiers. We show
empirically that our estimators are accurate and efficient for standard deep
learning models and demonstrate their usefulness for identifying vulnerable
data points, as well as quantifying robustness bias of models. Overall, our
tools provide a complementary view to robustness, improving our ability to
characterize model behaviour.
