  We introduce ImageNot, a dataset designed to match the scale of ImageNet
while differing drastically in other aspects. We show that key model
architectures developed for ImageNet over the years rank identically when
trained and evaluated on ImageNot to how they rank on ImageNet. This is true
when training models from scratch or fine-tuning them. Moreover, the relative
improvements of each model over earlier models strongly correlate in both
datasets. We further give evidence that ImageNot has a similar utility as
ImageNet for transfer learning purposes. Our work demonstrates a surprising
degree of external validity in the relative performance of image classification
models. This stands in contrast with absolute accuracy numbers that typically
drop sharply even under small changes to a dataset.
