  Test-time adaptation (TTA) seeks to tackle potential distribution shifts
between training and testing data by adapting a given model w.r.t. any testing
sample. This task is particularly important for deep models when the test
environment changes frequently. Although some recent attempts have been made to
handle this task, we still face two practical challenges: 1) existing methods
have to perform backward computation for each test sample, resulting in
unbearable prediction cost to many applications; 2) while existing TTA
solutions can significantly improve the test performance on out-of-distribution
data, they often suffer from severe performance degradation on in-distribution
data after TTA (known as catastrophic forgetting). In this paper, we point out
that not all the test samples contribute equally to model adaptation, and
high-entropy ones may lead to noisy gradients that could disrupt the model.
Motivated by this, we propose an active sample selection criterion to identify
reliable and non-redundant samples, on which the model is updated to minimize
the entropy loss for test-time adaptation. Furthermore, to alleviate the
forgetting issue, we introduce a Fisher regularizer to constrain important
model parameters from drastic changes, where the Fisher importance is estimated
from test samples with generated pseudo labels. Extensive experiments on
CIFAR-10-C, ImageNet-C, and ImageNet-R verify the effectiveness of our proposed
method.
