  The ability to generalize under distributional shifts is essential to
reliable machine learning, while models optimized with empirical risk
minimization usually fail on non-$i.i.d$ testing data. Recently, invariant
learning methods for out-of-distribution (OOD) generalization propose to find
causally invariant relationships with multi-environments. However, modern
datasets are frequently multi-sourced without explicit source labels, rendering
many invariant learning methods inapplicable. In this paper, we propose
Kernelized Heterogeneous Risk Minimization (KerHRM) algorithm, which achieves
both the latent heterogeneity exploration and invariant learning in kernel
space, and then gives feedback to the original neural network by appointing
invariant gradient direction. We theoretically justify our algorithm and
empirically validate the effectiveness of our algorithm with extensive
experiments.
