  Test-time adaptation (TTA) has shown to be effective at tackling distribution
shifts between training and testing data by adapting a given model on test
samples. However, the online model updating of TTA may be unstable and this is
often a key obstacle preventing existing TTA methods from being deployed in the
real world. Specifically, TTA may fail to improve or even harm the model
performance when test data have: 1) mixed distribution shifts, 2) small batch
sizes, and 3) online imbalanced label distribution shifts, which are quite
common in practice. In this paper, we investigate the unstable reasons and find
that the batch norm layer is a crucial factor hindering TTA stability.
Conversely, TTA can perform more stably with batch-agnostic norm layers, \ie,
group or layer norm. However, we observe that TTA with group and layer norms
does not always succeed and still suffers many failure cases. By digging into
the failure cases, we find that certain noisy test samples with large gradients
may disturb the model adaption and result in collapsed trivial solutions, \ie,
assigning the same class label for all samples. To address the above collapse
issue, we propose a sharpness-aware and reliable entropy minimization method,
called SAR, for further stabilizing TTA from two aspects: 1) remove partial
noisy samples with large gradients, 2) encourage model weights to go to a flat
minimum so that the model is robust to the remaining noisy samples. Promising
results demonstrate that SAR performs more stably over prior methods and is
computationally efficient under the above wild test scenarios.
