  Unsupervised Domain Adaptation (UDA) aims to leverage a label-rich source
domain to solve tasks on a related unlabeled target domain. It is a challenging
problem especially when a large domain gap lies between the source and target
domains. In this paper we propose a novel solution named SSRT (Safe
Self-Refinement for Transformer-based domain adaptation), which brings
improvement from two aspects. First, encouraged by the success of vision
transformers in various vision tasks, we arm SSRT with a transformer backbone.
We find that the combination of vision transformer with simple adversarial
adaptation surpasses best reported Convolutional Neural Network (CNN)-based
results on the challenging DomainNet benchmark, showing its strong transferable
feature representation. Second, to reduce the risk of model collapse and
improve the effectiveness of knowledge transfer between domains with large
gaps, we propose a Safe Self-Refinement strategy. Specifically, SSRT utilizes
predictions of perturbed target domain data to refine the model. Since the
model capacity of vision transformer is large and predictions in such
challenging tasks can be noisy, a safe training mechanism is designed to
adaptively adjust learning configuration. Extensive evaluations are conducted
on several widely tested UDA benchmarks and SSRT achieves consistently the best
performances, including 85.43% on Office-Home, 88.76% on VisDA-2017 and 45.2%
on DomainNet.
