  Adversarial domain adaptation has made impressive advances in transferring
knowledge from the source domain to the target domain by aligning feature
distributions of both domains. These methods focus on minimizing domain
divergence and regard the adaptability, which is measured as the expected error
of the ideal joint hypothesis on these two domains, as a small constant.
However, these approaches still face two issues: (1) Adversarial domain
alignment distorts the original feature distributions, deteriorating the
adaptability; (2) Transforming feature representations to be domain-invariant
needs to sacrifice domain-specific variations, resulting in weaker
discriminability. In order to alleviate these issues, we propose
category-invariant feature enhancement (CIFE), a general mechanism that
enhances the adversarial domain adaptation through optimizing the adaptability.
Specifically, the CIFE approach introduces category-invariant features to boost
the discriminability of domain-invariant features with preserving the
transferability. Experiments show that the CIFE could improve upon
representative adversarial domain adaptation methods to yield state-of-the-art
results on five benchmarks.
