  Over the past decade, the machine learning security community has developed a
myriad of defenses for evasion attacks. An understudied question in that
community is: for whom do these defenses defend? This work considers common
approaches to defending learned systems and how security defenses result in
performance inequities across different sub-populations. We outline appropriate
parity metrics for analysis and begin to answer this question through empirical
results of the fairness implications of machine learning security methods. We
find that many methods that have been proposed can cause direct harm, like
false rejection and unequal benefits from robustness training. The framework we
propose for measuring defense equality can be applied to robustly trained
models, preprocessing-based defenses, and rejection methods. We identify a set
of datasets with a user-centered application and a reasonable computational
cost suitable for case studies in measuring the equality of defenses. In our
case study of speech command recognition, we show how such adversarial training
and augmentation have non-equal but complex protections for social subgroups
across gender, accent, and age in relation to user coverage. We present a
comparison of equality between two rejection-based defenses: randomized
smoothing and neural rejection, finding randomized smoothing more equitable due
to the sampling mechanism for minority groups. This represents the first work
examining the disparity in the adversarial robustness in the speech domain and
the fairness evaluation of rejection-based defenses.
