  Practical real world datasets with plentiful categories introduce new
challenges for unsupervised domain adaptation like small inter-class
discriminability, that existing approaches relying on domain invariance alone
cannot handle sufficiently well. In this work we propose MemSAC, which exploits
sample level similarity across source and target domains to achieve
discriminative transfer, along with architectures that scale to a large number
of categories. For this purpose, we first introduce a memory augmented approach
to efficiently extract pairwise similarity relations between labeled source and
unlabeled target domain instances, suited to handle an arbitrary number of
classes. Next, we propose and theoretically justify a novel variant of the
contrastive loss to promote local consistency among within-class cross domain
samples while enforcing separation between classes, thus preserving
discriminative transfer from source to target. We validate the advantages of
MemSAC with significant improvements over previous state-of-the-art on multiple
challenging transfer tasks designed for large-scale adaptation, such as
DomainNet with 345 classes and fine-grained adaptation on Caltech-UCSD birds
dataset with 200 classes. We also provide in-depth analysis and insights into
the effectiveness of MemSAC.
