  Covariate-shift generalization, a typical case in out-of-distribution (OOD)
generalization, requires a good performance on the unknown test distribution,
which varies from the accessible training distribution in the form of covariate
shift. Recently, independence-driven importance weighting algorithms in stable
learning literature have shown empirical effectiveness to deal with
covariate-shift generalization on several learning models, including regression
algorithms and deep neural networks, while their theoretical analyses are
missing. In this paper, we theoretically prove the effectiveness of such
algorithms by explaining them as feature selection processes. We first specify
a set of variables, named minimal stable variable set, that is the minimal and
optimal set of variables to deal with covariate-shift generalization for common
loss functions, such as the mean squared loss and binary cross-entropy loss.
Afterward, we prove that under ideal conditions, independence-driven importance
weighting algorithms could identify the variables in this set. Analysis of
asymptotic properties is also provided. These theories are further validated in
several synthetic experiments.
