  Self-supervised learning (SSL) models reshaped our approach to speech,
language and vision. However their huge size and the opaque relations between
their layers and tasks result in slow inference and network overthinking, where
predictions made from the last layer of large models is worse than those made
from intermediate layers. Early exit (EE) strategies can solve both issues by
dynamically reducing computations at inference time for certain samples.
Although popular for classification tasks in vision and language, EE has seen
less use for sequence-to-sequence speech recognition (ASR) tasks where outputs
from early layers are often degenerate. This challenge is further compounded
when speech SSL models are applied on out-of-distribution (OOD) data. This
paper first shows that SSL models do overthinking in ASR. We then motivate
further research in EE by computing an optimal bound for performance versus
speed trade-offs. To approach this bound we propose two new strategies for ASR:
(1) we adapt the recently proposed patience strategy to ASR; and (2) we design
a new EE strategy specific to ASR that performs better than all strategies
previously introduced.
