  We consider representation learning (hypothesis class $\mathcal{H} =
\mathcal{F}\circ\mathcal{G}$) where training and test distributions can be
different. Recent studies provide hints and failure examples for domain
invariant representation learning, a common approach for this problem, but the
explanations provided are somewhat different and do not provide a unified
picture. In this paper, we provide new decompositions of risk which give
finer-grained explanations and clarify potential generalization issues. For
Single-Source Domain Adaptation, we give an exact decomposition (an equality)
of the target risk, via a natural hybrid argument, as sum of three factors: (1)
source risk, (2) representation conditional label divergence, and (3)
representation covariate shift. We derive a similar decomposition for the
Multi-Source case. These decompositions reveal factors (2) and (3) as the
precise reasons for failure to generalize. For example, we demonstrate that
domain adversarial neural networks (DANN) attempt to regularize for (3) but
miss (2), while a recent technique Invariant Risk Minimization (IRM) attempts
to account for (2) but does not consider (3). We also verify our observations
experimentally.
