  Recent works demonstrate that early layers in a neural network contain useful
information for prediction. Inspired by this, we show that extending
temperature scaling across all layers improves both calibration and accuracy.
We call this procedure "layer-stack temperature scaling" (LATES). Informally,
LATES grants each layer a weighted vote during inference. We evaluate it on
five popular convolutional neural network architectures both in- and
out-of-distribution and observe a consistent improvement over temperature
scaling in terms of accuracy, calibration, and AUC. All conclusions are
supported by comprehensive statistical analyses. Since LATES neither retrains
the architecture nor introduces many more parameters, its advantages can be
reaped without requiring additional data beyond what is used in temperature
scaling. Finally, we show that combining LATES with Monte Carlo Dropout matches
state-of-the-art results on CIFAR10/100.
