  The robustness of deep neural networks is usually lacking under adversarial
examples, common corruptions, and distribution shifts, which becomes an
important research problem in the development of deep learning. Although new
deep learning methods and robustness improvement techniques have been
constantly proposed, the robustness evaluations of existing methods are often
inadequate due to their rapid development, diverse noise patterns, and simple
evaluation metrics. Without thorough robustness evaluations, it is hard to
understand the advances in the field and identify the effective methods. In
this paper, we establish a comprehensive robustness benchmark called
\textbf{ARES-Bench} on the image classification task. In our benchmark, we
evaluate the robustness of 55 typical deep learning models on ImageNet with
diverse architectures (e.g., CNNs, Transformers) and learning algorithms (e.g.,
normal supervised training, pre-training, adversarial training) under numerous
adversarial attacks and out-of-distribution (OOD) datasets. Using robustness
curves as the major evaluation criteria, we conduct large-scale experiments and
draw several important findings, including: 1) there is an inherent trade-off
between adversarial and natural robustness for the same model architecture; 2)
adversarial training effectively improves adversarial robustness, especially
when performed on Transformer architectures; 3) pre-training significantly
improves natural robustness based on more training data or self-supervised
learning. Based on ARES-Bench, we further analyze the training tricks in
large-scale adversarial training on ImageNet. By designing the training
settings accordingly, we achieve the new state-of-the-art adversarial
robustness. We have made the benchmarking results and code platform publicly
available.
