  Recent research on robustness has revealed significant performance gaps
between neural image classifiers trained on datasets that are similar to the
test set, and those that are from a naturally shifted distribution, such as
sketches, paintings, and animations of the object categories observed during
training. Prior work focuses on reducing this gap by designing engineered
augmentations of training data or through unsupervised pretraining of a single
large model on massive in-the-wild training datasets scraped from the Internet.
However, the notion of a dataset is also undergoing a paradigm shift in recent
years. With drastic improvements in the quality, ease-of-use, and access to
modern generative models, generated data is pervading the web. In this light,
we study the question: How do these generated datasets influence the natural
robustness of image classifiers? We find that Imagenet classifiers trained on
real data augmented with generated data achieve higher accuracy and effective
robustness than standard training and popular augmentation strategies in the
presence of natural distribution shifts. We analyze various factors influencing
these results, including the choice of conditioning strategies and the amount
of generated data. Additionally, we find that the standard ImageNet classifiers
suffer a performance degradation of upto 20\% on the generated data, indicating
their fragility at accurately classifying the objects under novel variations.
Lastly, we demonstrate that the image classifiers, which have been trained on
real data augmented with generated data from the base generative model, exhibit
greater resilience to natural distribution shifts compared to the classifiers
trained on real data augmented with generated data from the finetuned
generative model on the real data. The code, models, and datasets are available
at https://github.com/Hritikbansal/generative-robustness.
