  Modern image classifiers are very accurate, but the predictions come without
uncertainty estimates. Conformal predictors provide uncertainty estimates by
computing a set of classes containing the correct class with a user-specified
probability based on the classifier's probability estimates. To provide such
sets, conformal predictors often estimate a cutoff threshold for the
probability estimates based on a calibration set. Conformal predictors
guarantee reliability only when the calibration set is from the same
distribution as the test set. Therefore, conformal predictors need to be
recalibrated for new distributions. However, in practice, labeled data from new
distributions is rarely available, making calibration infeasible. In this work,
we consider the problem of predicting the cutoff threshold for a new
distribution based on unlabeled examples. While it is impossible in general to
guarantee reliability when calibrating based on unlabeled examples, we propose
a method that provides excellent uncertainty estimates under natural
distribution shifts, and provably works for a specific model of a distribution
shift.
