  Fully test-time adaptation aims at adapting a pre-trained model to the test
stream during real-time inference, which is urgently required when the test
distribution differs from the training distribution. Several efforts have been
devoted to improving adaptation performance. However, we find that two
unfavorable defects are concealed in the prevalent adaptation methodologies
like test-time batch normalization (BN) and self-learning. First, we reveal
that the normalization statistics in test-time BN are completely affected by
the currently received test samples, resulting in inaccurate estimates. Second,
we show that during test-time adaptation, the parameter update is biased
towards some dominant classes. In addition to the extensively studied test
stream with independent and class-balanced samples, we further observe that the
defects can be exacerbated in more complicated test environments, such as
(time) dependent or class-imbalanced data. We observe that previous approaches
work well in certain scenarios while show performance degradation in others due
to their faults. In this paper, we provide a plug-in solution called DELTA for
Degradation-freE fuLly Test-time Adaptation, which consists of two components:
(i) Test-time Batch Renormalization (TBR), introduced to improve the estimated
normalization statistics. (ii) Dynamic Online re-weighTing (DOT), designed to
address the class bias within optimization. We investigate various test-time
adaptation methods on three commonly used datasets with four scenarios, and a
newly introduced real-world dataset. DELTA can help them deal with all
scenarios simultaneously, leading to SOTA performance.
