  Standard training datasets for deep learning often contain objects in common
settings (e.g., "a horse on grass" or "a ship in water") since they are usually
collected by randomly scraping the web. Uncommon and rare settings (e.g., "a
plane on water", "a car in snowy weather") are thus severely under-represented
in the training data. This can lead to an undesirable bias in model predictions
towards common settings and create a false sense of accuracy. In this paper, we
introduce FOCUS (Familiar Objects in Common and Uncommon Settings), a dataset
for stress-testing the generalization power of deep image classifiers. By
leveraging the power of modern search engines, we deliberately gather data
containing objects in common and uncommon settings in a wide range of
locations, weather conditions, and time of day. We present a detailed analysis
of the performance of various popular image classifiers on our dataset and
demonstrate a clear drop in performance when classifying images in uncommon
settings. By analyzing deep features of these models, we show that such errors
can be due to the use of spurious features in model predictions. We believe
that our dataset will aid researchers in understanding the inability of deep
models to generalize well to uncommon settings and drive future work on
improving their distributional robustness.
