  Research has shown that deep networks tend to be overly optimistic about
their predictions, leading to an underestimation of prediction errors. Due to
the limited nature of data, existing studies have proposed various methods
based on model prediction probabilities to bin the data and evaluate
calibration error. We propose a more generalized definition of calibration
error called Partitioned Calibration Error (PCE), revealing that the key
difference among these calibration error metrics lies in how the data space is
partitioned. We put forth an intuitive proposition that an accurate model
should be calibrated across any partition, suggesting that the input space
partitioning can extend beyond just the partitioning of prediction
probabilities, and include partitions directly related to the input. Through
semantic-related partitioning functions, we demonstrate that the relationship
between model accuracy and calibration lies in the granularity of the
partitioning function. This highlights the importance of partitioning criteria
for training a calibrated and accurate model. To validate the aforementioned
analysis, we propose a method that involves jointly learning a semantic aware
grouping function based on deep model features and logits to partition the data
space into subsets. Subsequently, a separate calibration function is learned
for each subset. Experimental results demonstrate that our approach achieves
significant performance improvements across multiple datasets and network
architectures, thus highlighting the importance of the partitioning function
for calibration.
