  Which parts of a dataset will a given model find difficult? Recent work has
shown that SGD-trained models have a bias towards simplicity, leading them to
prioritize learning a majority class, or to rely upon harmful spurious
correlations. Here, we show that the preference for "easy" runs far deeper: A
model may prioritize any class or group of the dataset that it finds simple-at
the expense of what it finds complex-as measured by performance difference on
the test set. When subsets with different levels of complexity align with
demographic groups, we term this difficulty disparity, a phenomenon that occurs
even with balanced datasets that lack group/label associations. We show how
difficulty disparity is a model-dependent quantity, and is further amplified in
commonly-used models as selected by typical average performance scores. We
quantify an amplification factor across a range of settings in order to compare
disparity of different models on a fixed dataset. Finally, we present two
real-world examples of difficulty amplification in action, resulting in
worse-than-expected performance disparities between groups even when using a
balanced dataset. The existence of such disparities in balanced datasets
demonstrates that merely balancing sample sizes of groups is not sufficient to
ensure unbiased performance. We hope this work presents a step towards
measurable understanding of the role of model bias as it interacts with the
structure of data, and call for additional model-dependent mitigation methods
to be deployed alongside dataset audits.
