  Many techniques have been developed, such as model compression, to make Deep
Neural Networks (DNNs) inference more efficiently. Nevertheless, DNNs still
lack excellent run-time dynamic inference capability to enable users trade-off
accuracy and computation complexity (i.e., latency on target hardware) after
model deployment, based on dynamic requirements and environments. Such research
direction recently draws great attention, where one realization is to train the
target DNN through a multiple-term objective function, which consists of
cross-entropy terms from multiple sub-nets. Our investigation in this work show
that the performance of dynamic inference highly relies on the quality of
sub-net sampling. With objective to construct a dynamic DNN and search multiple
high quality sub-nets with minimal searching cost, we propose a progressive
sub-net searching framework, which is embedded with several effective
techniques, including trainable noise ranking, channel group and fine-tuning
threshold setting, sub-nets re-selection. The proposed framework empowers the
target DNN with better dynamic inference capability, which outperforms prior
works on both CIFAR-10 and ImageNet dataset via comprehensive experiments on
different network structures. Taken ResNet18 as an example, our proposed method
achieves much better dynamic inference accuracy compared with prior popular
Universally-Slimmable-Network by 4.4%-maximally and 2.3%-averagely in ImageNet
dataset with the same model size.
