  Multi-exit network is a promising architecture for efficient model inference
by sharing backbone networks and weights among multiple exits. However, the
gradient conflict of the shared weights results in sub-optimal accuracy. This
paper introduces Deep Feature Surgery (\methodname), which consists of feature
partitioning and feature referencing approaches to resolve gradient conflict
issues during the training of multi-exit networks. The feature partitioning
separates shared features along the depth axis among all exits to alleviate
gradient conflict while simultaneously promoting joint optimization for each
exit. Subsequently, feature referencing enhances multi-scale features for
distinct exits across varying depths to improve the model accuracy.
Furthermore, \methodname~reduces the training operations with the reduced
complexity of backpropagation. Experimental results on Cifar100 and ImageNet
datasets exhibit that \methodname~provides up to a \textbf{50.00\%} reduction
in training time and attains up to a \textbf{6.94\%} enhancement in accuracy
when contrasted with baseline methods across diverse models and tasks. Budgeted
batch classification evaluation on MSDNet demonstrates that DFS uses about
$\mathbf{2}\boldsymbol{\times}$ fewer average FLOPs per image to achieve the
same classification accuracy as baseline methods on Cifar100.
