  Distribution shift (DS) may have two levels: the distribution itself changes,
and the support (i.e., the set where the probability density is non-zero) also
changes. When considering the support change between the training and test
distributions, there can be four cases: (i) they exactly match; (ii) the
training support is wider (and thus covers the test support); (iii) the test
support is wider; (iv) they partially overlap. Existing methods are good at
cases (i) and (ii), while cases (iii) and (iv) are more common nowadays but
still under-explored. In this paper, we generalize importance weighting (IW), a
golden solver for cases (i) and (ii), to a universal solver for all cases.
Specifically, we first investigate why IW might fail in cases (iii) and (iv);
based on the findings, we propose generalized IW (GIW) that could handle cases
(iii) and (iv) and would reduce to IW in cases (i) and (ii). In GIW, the test
support is split into an in-training (IT) part and an out-of-training (OOT)
part, and the expected risk is decomposed into a weighted classification term
over the IT part and a standard classification term over the OOT part, which
guarantees the risk consistency of GIW. Then, the implementation of GIW
consists of three components: (a) the split of validation data is carried out
by the one-class support vector machine, (b) the first term of the empirical
risk can be handled by any IW algorithm given training data and IT validation
data, and (c) the second term just involves OOT validation data. Experiments
demonstrate that GIW is a universal solver for DS problems, outperforming IW
methods in cases (iii) and (iv).
