  One challenge of object recognition is to generalize to new domains, to more
classes and/or to new modalities. This necessitates methods to combine and
reuse existing datasets that may belong to different domains, have partial
annotations, and/or have different data modalities. This paper formulates this
as a multi-source domain adaptation and label unification problem, and proposes
a novel method for it. Our method consists of a partially-supervised adaptation
stage and a fully-supervised adaptation stage. In the former, partial knowledge
is transferred from multiple source domains to the target domain and fused
therein. Negative transfer between unmatching label spaces is mitigated via
three new modules: domain attention, uncertainty maximization and
attention-guided adversarial alignment. In the latter, knowledge is transferred
in the unified label space after a label completion process with pseudo-labels.
Extensive experiments on three different tasks - image classification, 2D
semantic image segmentation, and joint 2D-3D semantic segmentation - show that
our method outperforms all competing methods significantly.
