  We expect the generalization error to improve with more samples from a
similar task, and to deteriorate with more samples from an out-of-distribution
(OOD) task. In this work, we show a counter-intuitive phenomenon: the
generalization error of a task can be a non-monotonic function of the number of
OOD samples. As the number of OOD samples increases, the generalization error
on the target task improves before deteriorating beyond a threshold. In other
words, there is value in training on small amounts of OOD data. We use Fisher's
Linear Discriminant on synthetic datasets and deep networks on computer vision
benchmarks such as MNIST, CIFAR-10, CINIC-10, PACS and DomainNet to demonstrate
and analyze this phenomenon. In the idealistic setting where we know which
samples are OOD, we show that these non-monotonic trends can be exploited using
an appropriately weighted objective of the target and OOD empirical risk. While
its practical utility is limited, this does suggest that if we can detect OOD
samples, then there may be ways to benefit from them. When we do not know which
samples are OOD, we show how a number of go-to strategies such as
data-augmentation, hyper-parameter optimization, and pre-training are not
enough to ensure that the target generalization error does not deteriorate with
the number of OOD samples in the dataset.
