  One of the primary challenges limiting the applicability of deep learning is
its susceptibility to learning spurious correlations rather than the underlying
mechanisms of the task of interest. The resulting failure to generalise cannot
be addressed by simply using more data from the same distribution. We propose
an auxiliary training objective that improves the generalization capabilities
of neural networks by leveraging an overlooked supervisory signal found in
existing datasets. We use pairs of minimally-different examples with different
labels, a.k.a counterfactual or contrasting examples, which provide a signal
indicative of the underlying causal structure of the task. We show that such
pairs can be identified in a number of existing datasets in computer vision
(visual question answering, multi-label image classification) and natural
language processing (sentiment analysis, natural language inference). The new
training objective orients the gradient of a model's decision function with
pairs of counterfactual examples. Models trained with this technique
demonstrate improved performance on out-of-distribution test sets.
