  For machine learning systems to be reliable, we must understand their
performance in unseen, out-of-distribution environments. In this paper, we
empirically show that out-of-distribution performance is strongly correlated
with in-distribution performance for a wide range of models and distribution
shifts. Specifically, we demonstrate strong correlations between
in-distribution and out-of-distribution performance on variants of CIFAR-10 &
ImageNet, a synthetic pose estimation task derived from YCB objects, satellite
imagery classification in FMoW-WILDS, and wildlife classification in
iWildCam-WILDS. The strong correlations hold across model architectures,
hyperparameters, training set size, and training duration, and are more precise
than what is expected from existing domain adaptation theory. To complete the
picture, we also investigate cases where the correlation is weaker, for
instance some synthetic distribution shifts from CIFAR-10-C and the tissue
classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory
based on a Gaussian data model that shows how changes in the data covariance
arising from distribution shift can affect the observed correlations.
