  We study linear regression under covariate shift, where the marginal
distribution over the input covariates differs in the source and the target
domains, while the conditional distribution of the output given the input
covariates is similar across the two domains. We investigate a transfer
learning approach with pretraining on the source data and finetuning based on
the target data (both conducted by online SGD) for this problem. We establish
sharp instance-dependent excess risk upper and lower bounds for this approach.
Our bounds suggest that for a large class of linear regression instances,
transfer learning with $O(N^2)$ source data (and scarce or no target data) is
as effective as supervised learning with $N$ target data. In addition, we show
that finetuning, even with only a small amount of target data, could
drastically reduce the amount of source data required by pretraining. Our
theory sheds light on the effectiveness and limitation of pretraining as well
as the benefits of finetuning for tackling covariate shift problems.
