  Most recent test-time adaptation methods focus on only classification tasks,
use specialized network architectures, destroy model calibration or rely on
lightweight information from the source domain. To tackle these issues, this
paper proposes a novel Test-time Self-Learning method with automatic
Adversarial augmentation dubbed TeSLA for adapting a pre-trained source model
to the unlabeled streaming test data. In contrast to conventional self-learning
methods based on cross-entropy, we introduce a new test-time loss function
through an implicitly tight connection with the mutual information and online
knowledge distillation. Furthermore, we propose a learnable efficient
adversarial augmentation module that further enhances online knowledge
distillation by simulating high entropy augmented images. Our method achieves
state-of-the-art classification and segmentation results on several benchmarks
and types of domain shifts, particularly on challenging measurement shifts of
medical images. TeSLA also benefits from several desirable properties compared
to competing methods in terms of calibration, uncertainty metrics,
insensitivity to model architectures, and source training strategies, all
supported by extensive ablations. Our code and models are available on GitHub.
