  In practical scenarios, it is often the case that the available training data
within the target domain only exist for a limited number of classes, with the
remaining classes only available within surrogate domains. We show that
including the target domain in training when there exist disjoint classes
between the target and surrogate domains creates significant negative transfer,
and causes performance to significantly decrease compared to training without
the target domain at all. We hypothesize that this negative transfer is due to
an intermediate shortcut that only occurs when multiple source domains are
present, and provide experimental evidence that this may be the case. We show
that this phenomena occurs on over 25 distinct domain shifts, both synthetic
and real, and in many cases deteriorates the performance to well worse than
random, even when using state-of-the-art domain adaptation methods.
