  Deep neural networks are powerful tools to detect hidden patterns in data and
leverage them to make predictions, but they are not designed to understand
uncertainty and estimate reliable probabilities. In particular, they tend to be
overconfident. We begin to address this problem in the context of multi-class
classification by developing a novel training algorithm producing models with
more dependable uncertainty estimates, without sacrificing predictive power.
The idea is to mitigate overconfidence by minimizing a loss function, inspired
by advances in conformal inference, that quantifies model uncertainty by
carefully leveraging hold-out data. Experiments with synthetic and real data
demonstrate this method can lead to smaller conformal prediction sets with
higher conditional coverage, after exact calibration with hold-out data,
compared to state-of-the-art alternatives.
