  Safely deploying machine learning models to the real world is often a
challenging process. Models trained with data obtained from a specific
geographic location tend to fail when queried with data obtained elsewhere,
agents trained in a simulation can struggle to adapt when deployed in the real
world or novel environments, and neural networks that are fit to a subset of
the population might carry some selection bias into their decision process. In
this work, we describe the problem of data shift from a novel
information-theoretic perspective by (i) identifying and describing the
different sources of error, (ii) comparing some of the most promising
objectives explored in the recent domain generalization, and fair
classification literature. From our theoretical analysis and empirical
evaluation, we conclude that the model selection procedure needs to be guided
by careful considerations regarding the observed data, the factors used for
correction, and the structure of the data-generating process.
