  The problem of attacks on neural networks through input modification (i.e.,
adversarial examples) has attracted much attention recently. Being relatively
easy to generate and hard to detect, these attacks pose a security breach that
many suggested defenses try to mitigate. However, the evaluation of the effect
of attacks and defenses commonly relies on traditional classification metrics,
without adequate adaptation to adversarial scenarios. Most of these metrics are
accuracy-based, and therefore may have a limited scope and low distinctive
power. Other metrics do not consider the unique characteristics of neural
networks functionality, or measure the effect of the attacks indirectly (e.g.,
through the complexity of their generation). In this paper, we present two
metrics which are specifically designed to measure the effect of attacks, or
the recovery effect of defenses, on the output of neural networks in multiclass
classification tasks. Inspired by the normalized discounted cumulative gain and
the reciprocal rank metrics used in information retrieval literature, we treat
the neural network predictions as ranked lists of results. Using additional
information about the probability of the rank enabled us to define novel
metrics that are suited to the task at hand. We evaluate our metrics using
various attacks and defenses on a pretrained VGG19 model and the ImageNet
dataset. Compared to the common classification metrics, our proposed metrics
demonstrate superior informativeness and distinctiveness.
