  Given a model trained on source data, Test-Time Adaptation (TTA) enables
adaptation and inference in test data streams with domain shifts from the
source. Current methods predominantly optimize the model for each incoming test
data batch using self-training loss. While these methods yield commendable
results in ideal test data streams, where batches are independently and
identically sampled from the target distribution, they falter under more
practical test data streams that are not independent and identically
distributed (non-i.i.d.). The data batches in a non-i.i.d. stream display
prominent label shifts relative to each other. It leads to conflicting
optimization objectives among batches during the TTA process. Given the
inherent risks of adapting the source model to unpredictable test-time
distributions, we reverse the adaptation process and propose a novel
Distribution Alignment loss for TTA. This loss guides the distributions of
test-time features back towards the source distributions, which ensures
compatibility with the well-trained source model and eliminates the pitfalls
associated with conflicting optimization objectives. Moreover, we devise a
domain shift detection mechanism to extend the success of our proposed TTA
method in the continual domain shift scenarios. Our extensive experiments
validate the logic and efficacy of our method. On six benchmark datasets, we
surpass existing methods in non-i.i.d. scenarios and maintain competitive
performance under the ideal i.i.d. assumption.
