  Uncertainty quantification is essential for the reliable deployment of
machine learning models to high-stakes application domains. Uncertainty
quantification is all the more challenging when training distribution and test
distribution are different, even the distribution shifts are mild. Despite the
ubiquity of distribution shifts in real-world applications, existing
uncertainty quantification approaches mainly study the in-distribution setting
where the train and test distributions are the same. In this paper, we develop
a systematic calibration model to handle distribution shifts by leveraging data
from multiple domains. Our proposed method -- multi-domain temperature scaling
-- uses the heterogeneity in the domains to improve calibration robustness
under distribution shift. Through experiments on three benchmark data sets, we
find our proposed method outperforms existing methods as measured on both
in-distribution and out-of-distribution test sets.
