  In many classification applications, the prediction of a deep neural network
(DNN) based classifier needs to be accompanied by some confidence indication.
Two popular approaches for that aim are: 1) Calibration: modifies the
classifier's softmax values such that the maximal value better estimates the
correctness probability; and 2) Conformal Prediction (CP): produces a
prediction set of candidate labels that contains the true label with a
user-specified probability, guaranteeing marginal coverage, rather than, e.g.,
per class coverage. In practice, both types of indications are desirable, yet,
so far the interplay between them has not been investigated. We start this
paper with an extensive empirical study of the effect of the popular
Temperature Scaling (TS) calibration on prominent CP methods and reveal that
while it improves the class-conditional coverage of adaptive CP methods,
surprisingly, it negatively affects their prediction set sizes. Subsequently,
we explore the effect of TS beyond its calibration application and offer simple
guidelines for practitioners to trade prediction set size and conditional
coverage of adaptive CP methods while effectively combining them with
calibration. Finally, we present a theoretical analysis of the effect of TS on
the prediction set sizes, revealing several mathematical properties of the
procedure, according to which we provide reasoning for this unintuitive
phenomenon.
