  Unsupervised domain adaptation (UDA) is one of the prominent tasks of
transfer learning, and it provides an effective approach to mitigate the
distribution shift between the labeled source domain and the unlabeled target
domain. Prior works mainly focus on aligning the marginal distributions or the
estimated class-conditional distributions. However, the joint dependency among
the feature and the label is crucial for the adaptation task and is not fully
exploited. To address this problem, we propose the Bures Joint Distribution
Alignment (BJDA) algorithm which directly models the joint distribution shift
based on the optimal transport theory in the infinite-dimensional kernel
spaces. Specifically, we propose a novel alignment loss term that minimizes the
kernel Bures-Wasserstein distance between the joint distributions. Technically,
BJDA can effectively capture the nonlinear structures underlying the data. In
addition, we introduce a dynamic margin in contrastive learning phase to
flexibly characterize the class separability and improve the discriminative
ability of representations. It also avoids the cross-validation procedure to
determine the margin parameter in traditional triplet loss based methods.
Extensive experiments show that BJDA is very effective for the UDA tasks, as it
outperforms state-of-the-art algorithms in most experimental settings. In
particular, BJDA improves the average accuracy of UDA tasks by 2.8% on
Adaptiope, 1.4% on Office-Caltech10, and 1.1% on ImageCLEF-DA.
