  Test-time adaptation is a special setting of unsupervised domain adaptation
where a trained model on the source domain has to adapt to the target domain
without accessing source data. We propose a novel way to leverage
self-supervised contrastive learning to facilitate target feature learning,
along with an online pseudo labeling scheme with refinement that significantly
denoises pseudo labels. The contrastive learning task is applied jointly with
pseudo labeling, contrasting positive and negative pairs constructed similarly
as MoCo but with source-initialized encoder, and excluding same-class negative
pairs indicated by pseudo labels. Meanwhile, we produce pseudo labels online
and refine them via soft voting among their nearest neighbors in the target
feature space, enabled by maintaining a memory queue. Our method, AdaContrast,
achieves state-of-the-art performance on major benchmarks while having several
desirable properties compared to existing works, including memory efficiency,
insensitivity to hyper-parameters, and better model calibration. Project page:
sites.google.com/view/adacontrast.
