  While standard Empirical Risk Minimization (ERM) training is proven effective
for image classification on in-distribution data, it fails to perform well on
out-of-distribution samples. One of the main sources of distribution shift for
image classification is the compositional nature of images. Specifically, in
addition to the main object or component(s) determining the label, some other
image components usually exist, which may lead to the shift of input
distribution between train and test environments. More importantly, these
components may have spurious correlations with the label. To address this
issue, we propose Decompose-and-Compose (DaC), which improves robustness to
correlation shift by a compositional approach based on combining elements of
images. Based on our observations, models trained with ERM usually highly
attend to either the causal components or the components having a high spurious
correlation with the label (especially in datapoints on which models have a
high confidence). In fact, according to the amount of spurious correlation and
the easiness of classification based on the causal or non-causal components,
the model usually attends to one of these more (on samples with high
confidence). Following this, we first try to identify the causal components of
images using class activation maps of models trained with ERM. Afterward, we
intervene on images by combining them and retraining the model on the augmented
data, including the counterfactual ones. Along with its high interpretability,
this work proposes a group-balancing method by intervening on images without
requiring group labels or information regarding the spurious features during
training. The method has an overall better worst group accuracy compared to
previous methods with the same amount of supervision on the group labels in
correlation shift.
