  As a simple technique to accelerate inference of large-scale pre-trained
models, early exiting has gained much attention in the NLP community. It allows
samples to exit early at internal classifiers without passing through the
entire model. Most existing work usually trains the internal classifiers
independently and employs an exiting strategy to decide whether or not to exit
based on the confidence of the current internal classifier. However, none of
these works takes full advantage of the fact that the internal classifiers are
trained to solve the same task therefore can be used to construct an ensemble.
In this paper, we show that a novel objective function for the training of the
ensemble internal classifiers can be naturally induced from the perspective of
ensemble learning and information theory. The proposed training objective
consists of two terms: one for accuracy and the other for the diversity of the
internal classifiers. In contrast, the objective used in prior work is exactly
the accuracy term of our training objective therefore only optimizes the
accuracy but not diversity. Further, we propose a simple voting-based strategy
that considers predictions of all the past internal classifiers to infer the
correct label and decide whether to exit. Experimental results on various NLP
tasks show that our proposed objective function and voting-based strategy can
achieve better accuracy-speed trade-offs.
