  We propose a framework for learning calibrated uncertainties under domain
shifts, where the source (training) distribution differs from the target (test)
distribution. We detect such domain shifts via a differentiable density ratio
estimator and train it together with the task network, composing an adjusted
softmax predictive form concerning domain shift. In particular, the density
ratio estimation reflects the closeness of a target (test) sample to the source
(training) distribution. We employ it to adjust the uncertainty of prediction
in the task network. This idea of using the density ratio is based on the
distributionally robust learning (DRL) framework, which accounts for the domain
shift by adversarial risk minimization. We show that our proposed method
generates calibrated uncertainties that benefit downstream tasks, such as
unsupervised domain adaptation (UDA) and semi-supervised learning (SSL). On
these tasks, methods like self-training and FixMatch use uncertainties to
select confident pseudo-labels for re-training. Our experiments show that the
introduction of DRL leads to significant improvements in cross-domain
performance. We also show that the estimated density ratios align with human
selection frequencies, suggesting a positive correlation with a proxy of human
perceived uncertainties.
