  Unsupervised domain adaptation enables intelligent models to transfer
knowledge from a labeled source domain to a similar but unlabeled target
domain. Recent study reveals that knowledge can be transferred from one source
domain to another unknown target domain, called Universal Domain Adaptation
(UDA). However, in the real-world application, there are often more than one
source domain to be exploited for domain adaptation. In this paper, we formally
propose a more general domain adaptation setting, universal multi-source domain
adaptation (UMDA), where the label sets of multiple source domains can be
different and the label set of target domain is completely unknown. The main
challenges in UMDA are to identify the common label set between each source
domain and target domain, and to keep the model scalable as the number of
source domains increases. To address these challenges, we propose a universal
multi-source adaptation network (UMAN) to solve the domain adaptation problem
without increasing the complexity of the model in various UMDA settings. In
UMAN, we estimate the reliability of each known class in the common label set
via the prediction margin, which helps adversarial training to better align the
distributions of multiple source domains and target domain in the common label
set. Moreover, the theoretical guarantee for UMAN is also provided. Massive
experimental results show that existing UDA and multi-source DA (MDA) methods
cannot be directly applied to UMDA and the proposed UMAN achieves the
state-of-the-art performance in various UMDA settings.
