  Dataset bias is a problem in adversarial machine learning, especially in the
evaluation of defenses. An adversarial attack or defense algorithm may show
better results on the reported dataset than can be replicated on other
datasets. Even when two algorithms are compared, their relative performance can
vary depending on the dataset. Deep learning offers state-of-the-art solutions
for image recognition, but deep models are vulnerable even to small
perturbations. Research in this area focuses primarily on adversarial attacks
and defense algorithms. In this paper, we report for the first time, a class of
robust images that are both resilient to attacks and that recover better than
random images under adversarial attacks using simple defense techniques. Thus,
a test dataset with a high proportion of robust images gives a misleading
impression about the performance of an adversarial attack or defense. We
propose three metrics to determine the proportion of robust images in a dataset
and provide scoring to determine the dataset bias. We also provide an
ImageNet-R dataset of 15000+ robust images to facilitate further research on
this intriguing phenomenon of image strength under attack. Our dataset,
combined with the proposed metrics, is valuable for unbiased benchmarking of
adversarial attack and defense algorithms.
