  Distribution shift over time occurs in many settings. Leveraging historical
data is necessary to learn a model for the last time point when limited data is
available in the final period, yet few methods have been developed specifically
for this purpose. In this work, we construct a benchmark with different
sequences of synthetic shifts to evaluate the effectiveness of 3 classes of
methods that 1) learn from all data without adapting to the final period, 2)
learn from historical data with no regard to the sequential nature and then
adapt to the final period, and 3) leverage the sequential nature of historical
data when tailoring a model to the final period. We call this benchmark
Seq-to-Final to highlight the focus on using a sequence of time periods to
learn a model for the final time point. Our synthetic benchmark allows users to
construct sequences with different types of shift and compare different
methods. We focus on image classification tasks using CIFAR-10 and CIFAR-100 as
the base images for the synthetic sequences. We also evaluate the same methods
on the Portraits dataset to explore the relevance to real-world shifts over
time. Finally, we create a visualization to contrast the initializations and
updates from different methods at the final time step. Our results suggest
that, for the sequences in our benchmark, methods that disregard the sequential
structure and adapt to the final time point tend to perform well. The
approaches we evaluate that leverage the sequential nature do not offer any
improvement. We hope that this benchmark will inspire the development of new
algorithms that are better at leveraging sequential historical data or a deeper
understanding of why methods that disregard the sequential nature are able to
perform well.
