  Slimmable Neural Networks (S-Net) is a novel network which enabled to select
one of the predefined proportions of channels (sub-network) dynamically
depending on the current computational resource availability. The accuracy of
each sub-network on S-Net, however, is inferior to that of individually trained
networks of the same size due to its difficulty of simultaneous optimization on
different sub-networks. In this paper, we propose Slimmable Pruned Neural
Networks (SP-Net), which has sub-network structures learned by pruning instead
of adopting structures with the same proportion of channels in each layer
(width multiplier) like S-Net, and we also propose new pruning procedures:
multi-base pruning instead of one-shot or iterative pruning to realize high
accuracy and huge training time saving. We also introduced slimmable channel
sorting (scs) to achieve calculation as fast as S-Net and zero padding match
(zpm) pruning to prune residual structure in more efficient way. SP-Net can be
combined with any kind of channel pruning methods and does not require any
complicated processing or time-consuming architecture search like NAS models.
Compared with each sub-network of the same FLOPs on S-Net, SP-Net improves
accuracy by 1.2-1.5% for ResNet-50, 0.9-4.4% for VGGNet, 1.3-2.7% for
MobileNetV1, 1.4-3.1% for MobileNetV2 on ImageNet. Furthermore, our methods
outperform other SOTA pruning methods and are on par with various NAS models
according to our experimental results on ImageNet. The code is available at
https://github.com/hideakikuratsu/SP-Net.
