  Uncertainty quantification is a central challenge in reliable and trustworthy
machine learning. Naive measures such as last-layer scores are well-known to
yield overconfident estimates in the context of overparametrized neural
networks. Several methods, ranging from temperature scaling to different
Bayesian treatments of neural networks, have been proposed to mitigate
overconfidence, most often supported by the numerical observation that they
yield better calibrated uncertainty measures. In this work, we provide a sharp
comparison between popular uncertainty measures for binary classification in a
mathematically tractable model for overparametrized neural networks: the random
features model. We discuss a trade-off between classification accuracy and
calibration, unveiling a double descent like behavior in the calibration curve
of optimally regularized estimators as a function of overparametrization. This
is in contrast with the empirical Bayes method, which we show to be well
calibrated in our setting despite the higher generalization error and
overparametrization.
