  In this paper, a unified approach is presented to transfer learning that
addresses several source and target domain label-space and annotation
assumptions with a single model. It is particularly effective in handling a
challenging case, where source and target label-spaces are disjoint, and
outperforms alternatives in both unsupervised and semi-supervised settings. The
key ingredient is a common representation termed Common Factorised Space. It is
shared between source and target domains, and trained with an unsupervised
factorisation loss and a graph-based loss. With a wide range of experiments, we
demonstrate the flexibility, relevance and efficacy of our method, both in the
challenging cases with disjoint label spaces, and in the more conventional
cases such as unsupervised domain adaptation, where the source and target
domains share the same label-sets.
