  Deploying neural networks to different devices or platforms is in general
challenging, especially when the model size is large or model complexity is
high. Although there exist ways for model pruning or distillation, it is
typically required to perform a full round of model training or finetuning
procedure in order to obtain a smaller model that satisfies the model size or
complexity constraints. Motivated by recent works on dynamic neural networks,
we propose a simple way to train a large network and flexibly extract a
subnetwork from it given a model size or complexity constraint during
inference. We introduce a new way to allow a large model to be trained with
dynamic depth and width during the training phase, and after the large model is
trained we can select a subnetwork from it with arbitrary depth and width
during the inference phase with a relatively better performance compared to
training the subnetwork independently from scratch. Experiment results on a
music source separation model show that our proposed method can effectively
improve the separation performance across different subnetwork sizes and
complexities with a single large model, and training the large model takes
significantly shorter time than training all the different subnetworks.
