  Predicting not only the target but also an accurate measure of uncertainty is
important for many machine learning applications and in particular
safety-critical ones. In this work we study the calibration of uncertainty
prediction for regression tasks which often arise in real-world systems. We
show that the existing definition for calibration of a regression uncertainty
[Kuleshov et al. 2018] has severe limitations in distinguishing informative
from non-informative uncertainty predictions. We propose a new definition that
escapes this caveat and an evaluation method using a simple histogram-based
approach. Our method clusters examples with similar uncertainty prediction and
compares the prediction with the empirical uncertainty on these examples. We
also propose a simple, scaling-based calibration method that preforms as well
as much more complex ones. We show results on both a synthetic, controlled
problem and on the object detection bounding-box regression task using the COCO
and KITTI datasets.
