  In the field of medical image analysis, achieving high accuracy is not
enough; ensuring well-calibrated predictions is also crucial. Confidence scores
of a deep neural network play a pivotal role in explainability by providing
insights into the model's certainty, identifying cases that require attention,
and establishing trust in its predictions. Consequently, the significance of a
well-calibrated model becomes paramount in the medical imaging domain, where
accurate and reliable predictions are of utmost importance. While there has
been a significant effort towards training modern deep neural networks to
achieve high accuracy on medical imaging tasks, model calibration and factors
that affect it remain under-explored. To address this, we conducted a
comprehensive empirical study that explores model performance and calibration
under different training regimes. We considered fully supervised training,
which is the prevailing approach in the community, as well as rotation-based
self-supervised method with and without transfer learning, across various
datasets and architecture sizes. Multiple calibration metrics were employed to
gain a holistic understanding of model calibration. Our study reveals that
factors such as weight distributions and the similarity of learned
representations correlate with the calibration trends observed in the models.
Notably, models trained using rotation-based self-supervised pretrained regime
exhibit significantly better calibration while achieving comparable or even
superior performance compared to fully supervised models across different
medical imaging datasets. These findings shed light on the importance of model
calibration in medical image analysis and highlight the benefits of
incorporating self-supervised learning approach to improve both performance and
calibration.
