  Despite recent advancements in deep learning, deep neural networks continue
to suffer from performance degradation when applied to new data that differs
from training data. Test-time adaptation (TTA) aims to address this challenge
by adapting a model to unlabeled data at test time. TTA can be applied to
pretrained networks without modifying their training procedures, enabling them
to utilize a well-formed source distribution for adaptation. One possible
approach is to align the representation space of test samples to the source
distribution (\textit{i.e.,} feature alignment). However, performing feature
alignment in TTA is especially challenging in that access to labeled source
data is restricted during adaptation. That is, a model does not have a chance
to learn test data in a class-discriminative manner, which was feasible in
other adaptation tasks (\textit{e.g.,} unsupervised domain adaptation) via
supervised losses on the source data. Based on this observation, we propose a
simple yet effective feature alignment loss, termed as Class-Aware Feature
Alignment (CAFA), which simultaneously 1) encourages a model to learn target
representations in a class-discriminative manner and 2) effectively mitigates
the distribution shifts at test time. Our method does not require any
hyper-parameters or additional losses, which are required in previous
approaches. We conduct extensive experiments on 6 different datasets and show
our proposed method consistently outperforms existing baselines.
