  The uncertainty estimation is critical in real-world decision making
applications, especially when distributional shift between the training and
test data are prevalent. Many calibration methods in the literature have been
proposed to improve the predictive uncertainty of DNNs which are generally not
well-calibrated. However, none of them is specifically designed to work
properly under domain shift condition. In this paper, we propose Unsupervised
Temperature Scaling (UTS) as a robust calibration method to domain shift. It
exploits unlabeled test samples instead of the training one to adjust the
uncertainty prediction of deep models towards the test distribution. UTS
utilizes a novel loss function, weighted NLL, which allows unsupervised
calibration. We evaluate UTS on a wide range of model-datasets to show the
possibility of calibration without labels and demonstrate the robustness of UTS
compared to other methods (e.g., TS, MC-dropout, SVI, ensembles) in shifted
domains.
