  Deep neural networks (DNNs) have achieved remarkable performance across a
wide range of applications, while they are vulnerable to adversarial examples,
which motivates the evaluation and benchmark of model robustness. However,
current evaluations usually use simple metrics to study the performance of
defenses, which are far from understanding the limitation and weaknesses of
these defense methods. Thus, most proposed defenses are quickly shown to be
attacked successfully, which results in the ``arm race'' phenomenon between
attack and defense. To mitigate this problem, we establish a model robustness
evaluation framework containing 23 comprehensive and rigorous metrics, which
consider two key perspectives of adversarial learning (i.e., data and model).
Through neuron coverage and data imperceptibility, we use data-oriented metrics
to measure the integrity of test examples; by delving into model structure and
behavior, we exploit model-oriented metrics to further evaluate robustness in
the adversarial setting. To fully demonstrate the effectiveness of our
framework, we conduct large-scale experiments on multiple datasets including
CIFAR-10, SVHN, and ImageNet using different models and defenses with our
open-source platform. Overall, our paper provides a comprehensive evaluation
framework, where researchers could conduct comprehensive and fast evaluations
using the open-source toolkit, and the analytical results could inspire deeper
understanding and further improvement to the model robustness.
