  We propose a unified look at jointly learning multiple vision tasks and
visual domains through universal representations, a single deep neural network.
Learning multiple problems simultaneously involves minimizing a weighted sum of
multiple loss functions with different magnitudes and characteristics and thus
results in unbalanced state of one loss dominating the optimization and poor
results compared to learning a separate model for each problem. To this end, we
propose distilling knowledge of multiple task/domain-specific networks into a
single deep neural network after aligning its representations with the
task/domain-specific ones through small capacity adapters. We rigorously show
that universal representations achieve state-of-the-art performances in
learning of multiple dense prediction problems in NYU-v2 and Cityscapes,
multiple image classification problems from diverse domains in Visual Decathlon
Dataset and cross-domain few-shot learning in MetaDataset. Finally we also
conduct multiple analysis through ablation and qualitative studies.
