  Multi-source Domain Adaptation (MDA) aims to transfer predictive models from
multiple, fully-labeled source domains to an unlabeled target domain. However,
in many applications, relevant labeled source datasets may not be available,
and collecting source labels can be as expensive as labeling the target data
itself. In this paper, we investigate Multi-source Few-shot Domain Adaptation
(MFDA): a new domain adaptation scenario with limited multi-source labels and
unlabeled target data. As we show, existing methods often fail to learn
discriminative features for both source and target domains in the MFDA setting.
Therefore, we propose a novel framework, termed Multi-Source Few-shot
Adaptation Network (MSFAN), which can be trained end-to-end in a
non-adversarial manner. MSFAN operates by first using a type of prototypical,
multi-domain, self-supervised learning to learn features that are not only
domain-invariant but also class-discriminative. Second, MSFAN uses a small,
labeled support set to enforce feature consistency and domain invariance across
domains. Finally, prototypes from multiple sources are leveraged to learn
better classifiers. Compared with state-of-the-art MDA methods, MSFAN improves
the mean classification accuracy over different domain pairs on MFDA by 20.2%,
9.4%, and 16.2% on Office, Office-Home, and DomainNet, respectively.
