  Understanding the performance of machine learning (ML) models across diverse
data distributions is critically important for reliable applications. Despite
recent empirical studies positing a near-perfect linear correlation between
in-distribution (ID) and out-of-distribution (OOD) accuracies, we empirically
demonstrate that this correlation is more nuanced under subpopulation shifts.
Through rigorous experimentation and analysis across a variety of datasets,
models, and training epochs, we demonstrate that OOD performance often has a
nonlinear correlation with ID performance in subpopulation shifts. Our
findings, which contrast previous studies that have posited a linear
correlation in model performance during distribution shifts, reveal a "moon
shape" correlation (parabolic uptrend curve) between the test performance on
the majority subpopulation and the minority subpopulation. This non-trivial
nonlinear correlation holds across model architectures, hyperparameters,
training durations, and the imbalance between subpopulations. Furthermore, we
found that the nonlinearity of this "moon shape" is causally influenced by the
degree of spurious correlations in the training data. Our controlled
experiments show that stronger spurious correlation in the training data
creates more nonlinear performance correlation. We provide complementary
experimental and theoretical analyses for this phenomenon, and discuss its
implications for ML reliability and fairness. Our work highlights the
importance of understanding the nonlinear effects of model improvement on
performance in different subpopulations, and has the potential to inform the
development of more equitable and responsible machine learning models.
