  Unsupervised domain adaptation(UDA) and Source-free UDA(SFUDA) methods
formulate the problem involving two domains: source and target. They typically
employ a standard training approach that begins with models pre-trained on
large-scale datasets e.g., ImageNet, while rarely discussing its effect.
Recognizing this gap, we investigate the following research questions: (1) What
is the correlation among ImageNet, the source, and the target domain? (2) How
does pre-training on ImageNet influence the target risk? To answer the first
question, we empirically observed an interesting Spontaneous Pulling (SP)
Effect in fine-tuning where the discrepancies between any two of the three
domains (ImageNet, Source, Target) decrease but at the cost of the impaired
semantic structure of the pre-train domain. For the second question, we put
forward a theory to explain SP and quantify that the target risk is bound by
gradient disparities among the three domains. Our observations reveal a key
limitation of existing methods: it hinders the adaptation performance if the
semantic cluster structure of the pre-train dataset (i.e.ImageNet) is impaired.
To address it, we incorporate ImageNet as the third domain and redefine the
UDA/SFUDA as a three-player game. Specifically, inspired by the theory and
empirical findings, we present a novel framework termed TriDA which
additionally preserves the semantic structure of the pre-train dataset during
fine-tuning. Experimental results demonstrate that it achieves state-of-the-art
performance across various UDA and SFUDA benchmarks.
