  Distribution shifts are common in real-world datasets and can affect the
performance and reliability of deep learning models. In this paper, we study
two types of distribution shifts: diversity shifts, which occur when test
samples exhibit patterns unseen during training, and correlation shifts, which
occur when test data present a different correlation between seen invariant and
spurious features. We propose an integrated protocol to analyze both types of
shifts using datasets where they co-exist in a controllable manner. Finally, we
apply our approach to a real-world classification problem of skin cancer
analysis, using out-of-distribution datasets and specialized bias annotations.
Our protocol reveals three findings: 1) Models learn and propagate correlation
shifts even with low-bias training; this poses a risk of accumulating and
combining unaccountable weak biases; 2) Models learn robust features in high-
and low-bias scenarios but use spurious ones if test samples have them; this
suggests that spurious correlations do not impair the learning of robust
features; 3) Diversity shift can reduce the reliance on spurious correlations;
this is counter intuitive since we expect biased models to depend more on
biases when invariant features are missing. Our work has implications for
distribution shift research and practice, providing new insights into how
models learn and rely on spurious correlations under different types of shifts.
