  Existing domain adaptation (DA) usually assumes the domain shift comes from
either the covariates or the labels. However, in real-world applications,
samples selected from different domains could have biases in both the
covariates and the labels. In this paper, we propose a new assumption,
Factorizable Joint Shift (FJS), to handle the co-existence of sampling bias in
covariates and labels. Although allowing for the shift from both sides, FJS
assumes the independence of the bias between the two factors. We provide
theoretical and empirical understandings about when FJS degenerates to prior
assumptions and when it is necessary. We further propose Joint Importance
Aligning (JIA), a discriminative learning objective to obtain joint importance
estimators for both supervised and unsupervised domain adaptation. Our method
can be seamlessly incorporated with existing domain adaptation algorithms for
better importance estimation and weighting on the training data. Experiments on
a synthetic dataset demonstrate the advantage of our method.
