  Test-time adaptation (TTA) methods, which generally rely on the model's
predictions (e.g., entropy minimization) to adapt the source pretrained model
to the unlabeled target domain, suffer from noisy signals originating from 1)
incorrect or 2) open-set predictions. Long-term stable adaptation is hampered
by such noisy signals, so training models without such error accumulation is
crucial for practical TTA. To address these issues, including open-set TTA, we
propose a simple yet effective sample selection method inspired by the
following crucial empirical finding. While entropy minimization compels the
model to increase the probability of its predicted label (i.e., confidence
values), we found that noisy samples rather show decreased confidence values.
To be more specific, entropy minimization attempts to raise the confidence
values of an individual sample's prediction, but individual confidence values
may rise or fall due to the influence of signals from numerous other
predictions (i.e., wisdom of crowds). Due to this fact, noisy signals
misaligned with such 'wisdom of crowds', generally found in the correct
signals, fail to raise the individual confidence values of wrong samples,
despite attempts to increase them. Based on such findings, we filter out the
samples whose confidence values are lower in the adapted model than in the
original model, as they are likely to be noisy. Our method is widely applicable
to existing TTA methods and improves their long-term adaptation performance in
both image classification (e.g., 49.4% reduced error rates with TENT) and
semantic segmentation (e.g., 11.7% gain in mIoU with TENT).
