  Adaptive inference is a simple method for reducing inference costs. The
method works by maintaining multiple classifiers of different capacities, and
allocating resources to each test instance according to its difficulty. In this
work, we compare the two main approaches for adaptive inference, Early-Exit and
Multi-Model, when training data is limited. First, we observe that for models
with the same architecture and size, individual Multi-Model classifiers
outperform their Early-Exit counterparts by an average of 2.3%. We show that
this gap is caused by Early-Exit classifiers sharing model parameters during
training, resulting in conflicting gradient updates of model weights. We find
that despite this gap, Early-Exit still provides a better speed-accuracy
trade-off due to the overhead of the Multi-Model approach. To address these
issues, we propose SWEET (Separating Weights in Early Exit Transformers), an
Early-Exit fine-tuning method that assigns each classifier its own set of
unique model weights, not updated by other classifiers. We compare SWEET's
speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find
that it outperforms both methods at fast speeds while maintaining comparable
scores to Early-Exit at slow speeds. Moreover, SWEET individual classifiers
outperform Early-Exit ones by 1.1% on average. SWEET enjoys the benefits of
both methods, paving the way for further reduction of inference costs in NLP.
