  We develop and analyze a principled approach to kernel ridge regression under
covariate shift. The goal is to learn a regression function with small mean
squared error over a target distribution, based on unlabeled data from there
and labeled data that may have a different feature distribution. We propose to
split the labeled data into two subsets and conduct kernel ridge regression on
them separately to obtain a collection of candidate models and an imputation
model. We use the latter to fill the missing labels and then select the best
candidate model accordingly. Our non-asymptotic excess risk bounds show that in
quite general scenarios, our estimator adapts to the structure of the target
distribution as well as the covariate shift. It achieves the minimax optimal
error rate up to a logarithmic factor. The use of pseudo-labels in model
selection does not have major negative impacts.
