  Empirical studies suggest that machine learning models trained with empirical
risk minimization (ERM) often rely on attributes that may be spuriously
correlated with the class labels. Such models typically lead to poor
performance during inference for data lacking such correlations. In this work,
we explicitly consider a situation where potential spurious correlations are
present in the majority of training data. In contrast with existing approaches,
which use the ERM model outputs to detect the samples without spurious
correlations and either heuristically upweight or upsample those samples, we
propose the logit correction (LC) loss, a simple yet effective improvement on
the softmax cross-entropy loss, to correct the sample logit. We demonstrate
that minimizing the LC loss is equivalent to maximizing the group-balanced
accuracy, so the proposed LC could mitigate the negative impacts of spurious
correlations. Our extensive experimental results further reveal that the
proposed LC loss outperforms state-of-the-art solutions on multiple popular
benchmarks by a large margin, an average 5.5\% absolute improvement, without
access to spurious attribute labels. LC is also competitive with oracle methods
that make use of the attribute labels. Code is available at
https://github.com/shengliu66/LC.
