  Early-exiting dynamic neural networks (EDNN), as one type of dynamic neural
networks, has been widely studied recently. A typical EDNN has multiple
prediction heads at different layers of the network backbone. During inference,
the model will exit at either the last prediction head or an intermediate
prediction head where the prediction confidence is higher than a predefined
threshold. To optimize the model, these prediction heads together with the
network backbone are trained on every batch of training data. This brings a
train-test mismatch problem that all the prediction heads are optimized on all
types of data in training phase while the deeper heads will only see difficult
inputs in testing phase. Treating training and testing inputs differently at
the two phases will cause the mismatch between training and testing data
distributions. To mitigate this problem, we formulate an EDNN as an additive
model inspired by gradient boosting, and propose multiple training techniques
to optimize the model effectively. We name our method BoostNet. Our experiments
show it achieves the state-of-the-art performance on CIFAR100 and ImageNet
datasets in both anytime and budgeted-batch prediction modes. Our code is
released at https://github.com/SHI-Labs/Boosted-Dynamic-Networks.
