  To build robust, fair, and safe AI systems, we would like our classifiers to
say ``I don't know'' when facing test examples that are difficult or fall
outside of the training classes.The ubiquitous strategy to predict under
uncertainty is the simplistic \emph{reject-or-classify} rule: abstain from
prediction if epistemic uncertainty is high, classify otherwise.Unfortunately,
this recipe does not allow different sources of uncertainty to communicate with
each other, produces miscalibrated predictions, and it does not allow to
correct for misspecifications in our uncertainty estimates. To address these
three issues, we introduce \emph{unified uncertainty calibration (U2C)}, a
holistic framework to combine aleatoric and epistemic uncertainties. U2C
enables a clean learning-theoretical analysis of uncertainty estimation, and
outperforms reject-or-classify across a variety of ImageNet benchmarks. Our
code is available at:
https://github.com/facebookresearch/UnifiedUncertaintyCalibration
