  How to estimate the uncertainty of a given model is a crucial problem.
Current calibration techniques treat different classes equally and thus
implicitly assume that the distribution of training data is balanced, but
ignore the fact that real-world data often follows a long-tailed distribution.
In this paper, we explore the problem of calibrating the model trained from a
long-tailed distribution. Due to the difference between the imbalanced training
distribution and balanced test distribution, existing calibration methods such
as temperature scaling can not generalize well to this problem. Specific
calibration methods for domain adaptation are also not applicable because they
rely on unlabeled target domain instances which are not available. Models
trained from a long-tailed distribution tend to be more overconfident to head
classes. To this end, we propose a novel knowledge-transferring-based
calibration method by estimating the importance weights for samples of tail
classes to realize long-tailed calibration. Our method models the distribution
of each class as a Gaussian distribution and views the source statistics of
head classes as a prior to calibrate the target distributions of tail classes.
We adaptively transfer knowledge from head classes to get the target
probability density of tail classes. The importance weight is estimated by the
ratio of the target probability density over the source probability density.
Extensive experiments on CIFAR-10-LT, MNIST-LT, CIFAR-100-LT, and ImageNet-LT
datasets demonstrate the effectiveness of our method.
