  Test time adaptation is the process of adapting, in an unsupervised manner, a
pre-trained source model to each incoming batch of the test data (i.e., without
requiring a substantial portion of the test data to be available, as in
traditional domain adaptation) and without access to the source data. Since it
works with each batch of test data, it is well-suited for dynamic environments
where decisions need to be made as the data is streaming in. Current test time
adaptation methods are primarily focused on a single source model. We propose
the first completely unsupervised Multi-source Test Time Adaptation (MeTA)
framework that handles multiple source models and optimally combines them to
adapt to the test data. MeTA has two distinguishing features. First, it
efficiently obtains the optimal combination weights to combine the source
models to adapt to the test data distribution. Second, it identifies which of
the source model parameters to update so that only the model which is most
correlated to the target data is adapted, leaving the less correlated ones
untouched; this mitigates the issue of "forgetting" the source model parameters
by focusing only on the source model that exhibits the strongest correlation
with the test batch distribution. Experiments on diverse datasets demonstrate
that the combination of multiple source models does at least as well as the
best source (with hindsight knowledge), and performance does not degrade as the
test data distribution changes over time (robust to forgetting).
