  For domain generalization (DG) and unsupervised domain adaptation (UDA),
cross domain feature alignment has been widely explored to pull the feature
distributions of different domains in order to learn domain-invariant
representations. However, the feature alignment is in general task-ignorant and
could result in degradation of the discrimination power of the feature
representation and thus hinders the high performance. In this paper, we propose
a unified framework termed Feature Alignment and Restoration (FAR) to
simultaneously ensure high generalization and discrimination power of the
networks for effective DG and UDA. Specifically, we perform feature alignment
(FA) across domains by aligning the moments of the distributions of attentively
selected features to reduce their discrepancy. To ensure high discrimination,
we propose a Feature Restoration (FR) operation to distill task-relevant
features from the residual information and use them to compensate for the
aligned features. For better disentanglement, we enforce a dual ranking entropy
loss constraint in the FR step to encourage the separation of task-relevant and
task-irrelevant features. Extensive experiments on multiple classification
benchmarks demonstrate the high performance and strong generalization of our
FAR framework for both domain generalization and unsupervised domain
adaptation.
