  Slimmable networks are a family of neural networks that can instantly adjust
the runtime width. The width can be chosen from a predefined widths set to
adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we
propose a systematic approach to train universally slimmable networks
(US-Nets), extending slimmable networks to execute at arbitrary width, and
generalizing to networks both with and without batch normalization layers. We
further propose two improved training techniques for US-Nets, named the
sandwich rule and inplace distillation, to enhance training process and boost
testing accuracy. We show improved performance of universally slimmable
MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with
individually trained ones and 4-switch slimmable network baselines. We also
evaluate the proposed US-Nets and improved training techniques on tasks of
image super-resolution and deep reinforcement learning. Extensive ablation
experiments on these representative tasks demonstrate the effectiveness of our
proposed methods. Our discovery opens up the possibility to directly evaluate
FLOPs-Accuracy spectrum of network architectures. Code and models are available
at: https://github.com/JiahuiYu/slimmable_networks
