  Neural network-based decisions tend to be overconfident, where their raw
outcome probabilities do not align with the true decision probabilities.
Calibration of neural networks is an essential step towards more reliable deep
learning frameworks. Prior metrics of calibration error primarily utilize crisp
bin membership-based measures. This exacerbates skew in model probabilities and
portrays an incomplete picture of calibration error. In this work, we propose a
Fuzzy Calibration Error metric (FCE) that utilizes a fuzzy binning approach to
calculate calibration error. This approach alleviates the impact of probability
skew and provides a tighter estimate while measuring calibration error. We
compare our metric with ECE across different data populations and class
memberships. Our results show that FCE offers better calibration error
estimation, especially in multi-class settings, alleviating the effects of skew
in model confidence scores on calibration error estimation. We make our code
and supplementary materials available at: https://github.com/bihani-g/fce
