  In machine learning, it is commonly assumed that training and test data share
the same population distribution. However, this assumption is often violated in
practice because the sample selection bias may induce the distribution shift
from training data to test data. Such a model-agnostic distribution shift
usually leads to prediction instability across unknown test data. In this
paper, we propose a novel balance-subsampled stable prediction (BSSP) algorithm
based on the theory of fractional factorial design. It isolates the clear
effect of each predictor from the confounding variables. A design-theoretic
analysis shows that the proposed method can reduce the confounding effects
among predictors induced by the distribution shift, hence improve both the
accuracy of parameter estimation and prediction stability. Numerical
experiments on both synthetic and real-world data sets demonstrate that our
BSSP algorithm significantly outperforms the baseline methods for stable
prediction across unknown test data.
