  Dynamic neural networks are a recent technique that promises a remedy for the
increasing size of modern deep learning models by dynamically adapting their
computational cost to the difficulty of the inputs. In this way, the model can
adjust to a limited computational budget. However, the poor quality of
uncertainty estimates in deep learning models makes it difficult to distinguish
between hard and easy samples. To address this challenge, we present a
computationally efficient approach for post-hoc uncertainty quantification in
dynamic neural networks. We show that adequately quantifying and accounting for
both aleatoric and epistemic uncertainty through a probabilistic treatment of
the last layers improves the predictive performance and aids decision-making
when determining the computational budget. In the experiments, we show
improvements on CIFAR-100, ImageNet, and Caltech-256 in terms of accuracy,
capturing uncertainty, and calibration error.
