  Deep domain adaptation models learn a neural network in an unlabeled target
domain by leveraging the knowledge from a labeled source domain. This can be
achieved by learning a domain-invariant feature space. Though the learned
representations are separable in the source domain, they usually have a large
variance and samples with different class labels tend to overlap in the target
domain, which yields suboptimal adaptation performance. To fill the gap, a
Fisher loss is proposed to learn discriminative representations which are
within-class compact and between-class separable. Experimental results on two
benchmark datasets show that the Fisher loss is a general and effective loss
for deep domain adaptation. Noticeable improvements are brought when it is used
together with widely adopted transfer criteria, including MMD, CORAL and domain
adversarial loss. For example, an absolute improvement of 6.67% in terms of the
mean accuracy is attained when the Fisher loss is used together with the domain
adversarial loss on the Office-Home dataset.
