  Deep Ensembles are a simple, reliable, and effective method of improving both
the predictive performance and uncertainty estimates of deep learning
approaches. However, they are widely criticised as being computationally
expensive, due to the need to deploy multiple independent models. Recent work
has challenged this view, showing that for predictive accuracy, ensembles can
be more computationally efficient (at inference) than scaling single models
within an architecture family. This is achieved by cascading ensemble members
via an early-exit approach. In this work, we investigate extending these
efficiency gains to tasks related to uncertainty estimation. As many such
tasks, e.g. selective classification, are binary classification, our key novel
insight is to only pass samples within a window close to the binary decision
boundary to later cascade stages. Experiments on ImageNet-scale data across a
number of network architectures and uncertainty tasks show that the proposed
window-based early-exit approach is able to achieve a superior
uncertainty-computation trade-off compared to scaling single models. For
example, a cascaded EfficientNet-B2 ensemble is able to achieve similar
coverage at 5% risk as a single EfficientNet-B4 with <30% the number of MACs.
We also find that cascades/ensembles give more reliable improvements on OOD
data vs scaling models up. Code for this work is available at:
https://github.com/Guoxoug/window-early-exit.
