  The class-wise training losses often diverge as a result of the various
levels of intra-class and inter-class appearance variation, and we find that
the diverging class-wise training losses cause the uncalibrated prediction with
its reliability. To resolve the issue, we propose a new calibration method to
synchronize the class-wise training losses. We design a new training loss to
alleviate the variance of class-wise training losses by using multiple
class-wise scaling factors. Since our framework can compensate the training
losses of overfitted classes with those of under-fitted classes, the integrated
training loss is preserved, preventing the performance drop even after the
model calibration. Furthermore, our method can be easily employed in the
post-hoc calibration methods, allowing us to use the pre-trained model as an
initial model and reduce the additional computation for model calibration. We
validate the proposed framework by employing it in the various post-hoc
calibration methods, which generally improves calibration performance while
preserving accuracy, and discover through the investigation that our approach
performs well with unbalanced datasets and untuned hyperparameters.
