  In prediction tasks, there exist features that are related to the label in
the same way across different settings for that task; these are semantic
features or semantics. Features with varying relationships to the label are
nuisances. For example, in detecting cows from natural images, the shape of the
head is semantic but because images of cows often have grass backgrounds but
not always, the background is a nuisance. Models that exploit nuisance-label
relationships face performance degradation when these relationships change.
Building models robust to such changes requires additional knowledge beyond
samples of the features and labels. For example, existing work uses annotations
of nuisances or assumes ERM-trained models depend on nuisances. Approaches to
integrate new kinds of additional knowledge enlarge the settings where robust
models can be built. We develop an approach to use knowledge about the
semantics by corrupting them in data, and then using the corrupted data to
produce models which identify correlations between nuisances and the label.
Once these correlations are identified, they can be used to adjust for where
nuisances drive predictions. We study semantic corruptions in powering
different spurious-correlation avoiding methods on multiple out-of-distribution
(OOD) tasks like classifying waterbirds, natural language inference (NLI), and
detecting cardiomegaly in chest X-rays.
