  Artificial intelligence(AI)-assisted method had received much attention in
the risk field such as disease diagnosis. Different from the classification of
disease types, it is a fine-grained task to classify the medical images as
benign or malignant. However, most research only focuses on improving the
diagnostic accuracy and ignores the evaluation of model reliability, which
limits its clinical application. For clinical practice, calibration presents
major challenges in the low-data regime extremely for over-parametrized models
and inherent noises. In particular, we discovered that modeling data-dependent
uncertainty is more conducive to confidence calibrations. Compared with
test-time augmentation(TTA), we proposed a modified Bootstrapping loss(BS loss)
function with Mixup data augmentation strategy that can better calibrate
predictive uncertainty and capture data distribution transformation without
additional inference time. Our experiments indicated that BS loss with
Mixup(BSM) model can halve the Expected Calibration Error(ECE) compared to
standard data augmentation, deep ensemble and MC dropout. The correlation
between uncertainty and similarity of in-domain data is up to -0.4428 under the
BSM model. Additionally, the BSM model is able to perceive the semantic
distance of out-of-domain data, demonstrating high potential in real-world
clinical practice.
