  Overconfidence is a common issue for deep neural networks, limiting their
deployment in real-world applications. To better estimate confidence, existing
methods mostly focus on fully-supervised scenarios and rely on training labels.
In this paper, we propose the first confidence estimation method for a
semi-supervised setting, when most training labels are unavailable. We
stipulate that even with limited training labels, we can still reasonably
approximate the confidence of model on unlabeled samples by inspecting the
prediction consistency through the training process. We use training
consistency as a surrogate function and propose a consistency ranking loss for
confidence estimation. On both image classification and segmentation tasks, our
method achieves state-of-the-art performances in confidence estimation.
Furthermore, we show the benefit of the proposed method through a downstream
active learning task. The code is available at
https://github.com/TopoXLab/consistency-ranking-loss
