  Recent domain generalization (DG) approaches typically use the hypothesis
learned on source domains for inference on the unseen target domain. However,
such a hypothesis can be arbitrarily far from the optimal one for the target
domain, induced by a gap termed ``adaptivity gap''. Without exploiting the
domain information from the unseen test samples, adaptivity gap estimation and
minimization are intractable, which hinders us to robustify a model to any
unknown distribution. In this paper, we first establish a generalization bound
that explicitly considers the adaptivity gap. Our bound motivates two
strategies to reduce the gap: the first one is ensembling multiple classifiers
to enrich the hypothesis space, then we propose effective gap estimation
methods for guiding the selection of a better hypothesis for the target. The
other method is minimizing the gap directly by adapting model parameters using
online target samples. We thus propose \textbf{Domain-specific Risk
Minimization (DRM)}. During training, DRM models the distributions of different
source domains separately; for inference, DRM performs online model steering
using the source hypothesis for each arriving target sample. Extensive
experiments demonstrate the effectiveness of the proposed DRM for domain
generalization with the following advantages: 1) it significantly outperforms
competitive baselines on different distributional shift settings; 2) it
achieves either comparable or superior accuracies on all source domains
compared to vanilla empirical risk minimization; 3) it remains simple and
efficient during training, and 4) it is complementary to invariant learning
approaches.
