  Conventional unsupervised domain adaptation (UDA) studies the knowledge
transfer between a limited number of domains. This neglects the more practical
scenario where data are distributed in numerous different domains in the real
world. The domain similarity between those domains is critical for domain
adaptation performance. To describe and learn relations between different
domains, we propose a novel Domain2Vec model to provide vectorial
representations of visual domains based on joint learning of feature
disentanglement and Gram matrix. To evaluate the effectiveness of our
Domain2Vec model, we create two large-scale cross-domain benchmarks. The first
one is TinyDA, which contains 54 domains and about one million MNIST-style
images. The second benchmark is DomainBank, which is collected from 56 existing
vision datasets. We demonstrate that our embedding is capable of predicting
domain similarities that match our intuition about visual relations between
different domains. Extensive experiments are conducted to demonstrate the power
of our new datasets in benchmarking state-of-the-art multi-source domain
adaptation methods, as well as the advantage of our proposed model.
