  Humans are incredibly good at transferring knowledge from one domain to
another, enabling rapid learning of new tasks. Likewise, transfer learning has
enabled enormous success in many computer vision problems using pretraining.
However, the benefits of transfer in multi-domain learning, where a network
learns multiple tasks defined by different datasets, has not been adequately
studied. Learning multiple domains could be beneficial, or these domains could
interfere with each other given limited network capacity. Understanding how
deep neural networks of varied capacity facilitate transfer across inputs from
different distributions is a critical step towards open world learning. In this
work, we decipher the conditions where interference and knowledge transfer
occur in multi-domain learning. We propose new metrics disentangling
interference and transfer, set up experimental protocols, and examine the roles
of network capacity, task grouping, and dynamic loss weighting in reducing
interference and facilitating transfer.
