  Early exits are an important efficiency mechanism integrated into deep neural
networks that allows for the termination of the network's forward pass before
processing through all its layers. By allowing early halting of the inference
process for less complex inputs that reached high confidence, early exits
significantly reduce the amount of computation required. Early exit methods add
trainable internal classifiers which leads to more intricacy in the training
process. However, there is no consistent verification of the approaches of
training of early exit methods, and no unified scheme of training such models.
Most early exit methods employ a training strategy that either simultaneously
trains the backbone network and the exit heads or trains the exit heads
separately. We propose a training approach where the backbone is initially
trained on its own, followed by a phase where both the backbone and the exit
heads are trained together. Thus, we advocate for organizing early-exit
training strategies into three distinct categories, and then validate them for
their performance and efficiency. In this benchmark, we perform both
theoretical and empirical analysis of early-exit training regimes. We study the
methods in terms of information flow, loss landscape and numerical rank of
activations and gauge the suitability of regimes for various architectures and
datasets.
