  Dealing with distribution shifts is one of the central challenges for modern
machine learning. One fundamental situation is the covariate shift, where the
input distributions of data change from training to testing stages while the
input-conditional output distribution remains unchanged. In this paper, we
initiate the study of a more challenging scenario -- continuous covariate shift
-- in which the test data appear sequentially, and their distributions can
shift continuously. Our goal is to adaptively train the predictor such that its
prediction risk accumulated over time can be minimized. Starting with the
importance-weighted learning, we show the method works effectively if the
time-varying density ratios of test and train inputs can be accurately
estimated. However, existing density ratio estimation methods would fail due to
data scarcity at each time step. To this end, we propose an online method that
can appropriately reuse historical information. Our density ratio estimation
method is proven to perform well by enjoying a dynamic regret bound, which
finally leads to an excess risk guarantee for the predictor. Empirical results
also validate the effectiveness.
