  Confidence calibration is central to providing accurate and interpretable
uncertainty estimates, especially under safety-critical scenarios. However, we
find that existing calibration algorithms often overlook the issue of
*proximity bias*, a phenomenon where models tend to be more overconfident in
low proximity data (i.e., data lying in the sparse region of the data
distribution) compared to high proximity samples, and thus suffer from
inconsistent miscalibration across different proximity samples. We examine the
problem over 504 pretrained ImageNet models and observe that: 1) Proximity bias
exists across a wide variety of model architectures and sizes; 2)
Transformer-based models are relatively more susceptible to proximity bias than
CNN-based models; 3) Proximity bias persists even after performing popular
calibration algorithms like temperature scaling; 4) Models tend to overfit more
heavily on low proximity samples than on high proximity samples. Motivated by
the empirical findings, we propose ProCal, a plug-and-play algorithm with a
theoretical guarantee to adjust sample confidence based on proximity. To
further quantify the effectiveness of calibration algorithms in mitigating
proximity bias, we introduce proximity-informed expected calibration error
(PIECE) with theoretical analysis. We show that ProCal is effective in
addressing proximity bias and improving calibration on balanced, long-tail, and
distribution-shift settings under four metrics over various model
architectures. We believe our findings on proximity bias will guide the
development of *fairer and better-calibrated* models, contributing to the
broader pursuit of trustworthy AI. Our code is available at:
https://github.com/MiaoXiong2320/ProximityBias-Calibration.
