  Deep neural networks are increasingly utilized in various machine learning
tasks. However, as these models grow in complexity, they often face calibration
issues, despite enhanced prediction accuracy. Many studies have endeavored to
improve calibration performance through the use of specific loss functions,
data preprocessing and training frameworks. Yet, investigations into
calibration properties have been somewhat overlooked. Our study leverages the
Neural Architecture Search (NAS) search space, offering an exhaustive model
architecture space for thorough calibration properties exploration. We
specifically create a model calibration dataset. This dataset evaluates 90
bin-based and 12 additional calibration measurements across 117,702 unique
neural networks within the widely employed NATS-Bench search space. Our
analysis aims to answer several longstanding questions in the field, using our
proposed dataset: (i) Can model calibration be generalized across different
datasets? (ii) Can robustness be used as a calibration measurement? (iii) How
reliable are calibration metrics? (iv) Does a post-hoc calibration method
affect all models uniformly? (v) How does calibration interact with accuracy?
(vi) What is the impact of bin size on calibration measurement? (vii) Which
architectural designs are beneficial for calibration? Additionally, our study
bridges an existing gap by exploring calibration within NAS. By providing this
dataset, we enable further research into NAS calibration. As far as we are
aware, our research represents the first large-scale investigation into
calibration properties and the premier study of calibration issues within NAS.
The project page can be found at https://www.taolinwei.com/calibration-study
