  Multi-Exit models (MEMs) use an early-exit strategy to improve the accuracy
and efficiency of deep neural networks (DNNs) by allowing samples to exit the
network before the last layer. However, the effectiveness of MEMs in the
presence of distribution shifts remains largely unexplored. Our work examines
how distribution shifts generated by common image corruptions affect the
accuracy/efficiency of MEMs. We find that under common corruptions,
early-exiting at the first correct exit reduces the inference cost and provides
a significant boost in accuracy ( 10%) over exiting at the last layer. However,
with realistic early-exit strategies, which do not assume knowledge about the
correct exits, MEMs still reduce inference cost but provide a marginal
improvement in accuracy (1%) compared to exiting at the last layer. Moreover,
the presence of distribution shift widens the gap between an MEM's maximum
classification accuracy and realistic early-exit strategies by 5% on average
compared with the gap on in-distribution data. Our empirical analysis shows
that the lack of calibration due to a distribution shift increases the
susceptibility of such early-exit strategies to exit early and increases
misclassification rates. Furthermore, the lack of calibration increases the
inconsistency in the predictions of the model across exits, leading to both
inefficient inference and more misclassifications compared with evaluation on
in-distribution data. Finally, we propose two metrics, underthinking and
overthinking, that quantify the different behavior of practical early-exit
strategy under distribution shifts, and provide insights into improving the
practical utility of MEMs.
