  We investigate model assessment and selection in a changing environment, by
synthesizing datasets from both the current time period and historical epochs.
To tackle unknown and potentially arbitrary temporal distribution shift, we
develop an adaptive rolling window approach to estimate the generalization
error of a given model. This strategy also facilitates the comparison between
any two candidate models by estimating the difference of their generalization
errors. We further integrate pairwise comparisons into a single-elimination
tournament, achieving near-optimal model selection from a collection of
candidates. Theoretical analyses and numerical experiments demonstrate the
adaptivity of our proposed methods to the non-stationarity in data.
