  Current dataset collection methods typically scrape large amounts of data
from the web. While this technique is extremely scalable, data collected in
this way tends to reinforce stereotypical biases, can contain personally
identifiable information, and typically originates from Europe and North
America. In this work, we rethink the dataset collection paradigm and introduce
GeoDE, a geographically diverse dataset with 61,940 images from 40 classes and
6 world regions, and no personally identifiable information, collected through
crowd-sourcing. We analyse GeoDE to understand differences in images collected
in this manner compared to web-scraping. Despite the smaller size of this
dataset, we demonstrate its use as both an evaluation and training dataset,
highlight shortcomings in current models, as well as show improved performances
when even small amounts of GeoDE (1000 - 2000 images per region) are added to a
training dataset. We release the full dataset and code at
https://geodiverse-data-collection.cs.princeton.edu/
