  We investigate the robustness properties of ResNeXt class image recognition
models trained with billion scale weakly supervised data (ResNeXt WSL models).
These models, recently made public by Facebook AI, were trained with ~1B images
from Instagram and fine-tuned on ImageNet. We show that these models display an
unprecedented degree of robustness against common image corruptions and
perturbations, as measured by the ImageNet-C and ImageNet-P benchmarks. They
also achieve substantially improved accuracies on the recently introduced
"natural adversarial examples" benchmark (ImageNet-A). The largest of the
released models, in particular, achieves state-of-the-art results on
ImageNet-C, ImageNet-P, and ImageNet-A by a large margin. The gains on
ImageNet-C, ImageNet-P, and ImageNet-A far outpace the gains on ImageNet
validation accuracy, suggesting the former as more useful benchmarks to measure
further progress in image recognition. Remarkably, the ResNeXt WSL models even
achieve a limited degree of adversarial robustness against state-of-the-art
white-box attacks (10-step PGD attacks). However, in contrast to adversarially
trained models, the robustness of the ResNeXt WSL models rapidly declines with
the number of PGD steps, suggesting that these models do not achieve genuine
adversarial robustness. Visualization of the learned features also confirms
this conclusion. Finally, we show that although the ResNeXt WSL models are more
shape-biased than comparable ImageNet-trained models in a shape-texture cue
conflict experiment, they still remain much more texture-biased than humans,
suggesting that they share some of the underlying characteristics of
ImageNet-trained models that make this benchmark challenging.
