  Adversarial defenses are naturally evaluated on their ability to tolerate
adversarial attacks. To test defenses, diverse adversarial attacks are crafted,
that are usually described in terms of their evading capability and the L0, L1,
L2, and Linf norms. We question if the evading capability and L-norms are the
most effective information to claim that defenses have been tested against a
representative attack set. To this extent, we select image quality metrics from
the state of the art and search correlations between image perturbation and
detectability. We observe that computing L-norms alone is rarely the preferable
solution. We observe a strong correlation between the identified metrics
computed on an adversarial image and the output of a detector on such an image,
to the extent that they can predict the response of a detector with
approximately 0.94 accuracy. Further, we observe that metrics can classify
attacks based on similar perturbations and similar detectability. This suggests
a possible review of the approach to evaluate detectors, where additional
metrics are included to assure that a representative attack dataset is
selected.
