  Classifiers in machine learning are often brittle when deployed. Particularly
concerning are models with inconsistent performance on specific subgroups of a
class, e.g., exhibiting disparities in skin cancer classification in the
presence or absence of a spurious bandage. To mitigate these performance
differences, we introduce model patching, a two-stage framework for improving
robustness that encourages the model to be invariant to subgroup differences,
and focus on class information shared by subgroups. Model patching first models
subgroup features within a class and learns semantic transformations between
them, and then trains a classifier with data augmentations that deliberately
manipulate subgroup features. We instantiate model patching with CAMEL, which
(1) uses a CycleGAN to learn the intra-class, inter-subgroup augmentations, and
(2) balances subgroup performance using a theoretically-motivated subgroup
consistency regularizer, accompanied by a new robust objective. We demonstrate
CAMEL's effectiveness on 3 benchmark datasets, with reductions in robust error
of up to 33% relative to the best baseline. Lastly, CAMEL successfully patches
a model that fails due to spurious features on a real-world skin cancer
dataset.
