  In this work, we analyze the conditions under which information about the
context of an input $X$ can improve the predictions of deep learning models in
new domains. Following work in marginal transfer learning in Domain
Generalization (DG), we formalize the notion of context as a
permutation-invariant representation of a set of data points that originate
from the same domain as the input itself. We offer a theoretical analysis of
the conditions under which this approach can, in principle, yield benefits, and
formulate two necessary criteria that can be easily verified in practice.
Additionally, we contribute insights into the kind of distribution shifts for
which the marginal transfer learning approach promises robustness. Empirical
analysis shows that our criteria are effective in discerning both favorable and
unfavorable scenarios. Finally, we demonstrate that we can reliably detect
scenarios where a model is tasked with unwarranted extrapolation in
out-of-distribution (OOD) domains, identifying potential failure cases.
Consequently, we showcase a method to select between the most predictive and
the most robust model, circumventing the well-known trade-off between
predictive performance and robustness.
