  Deep neural networks (DNNs) must cater to a variety of users with different
performance needs and budgets, leading to the costly practice of training,
storing, and maintaining numerous user/task-specific models. There are
solutions in the literature to deal with single dynamic or many-in-one models
instead of many individual networks; however, they suffer from significant
drops in performance, lack of generalization across different model
architectures or different dimensions (e.g. depth, width, attention blocks),
heavy model search requirements during training, and training a limited number
of sub-models. To address these limitations, we propose SortedNet, a
generalized and scalable training solution to harness the inherent modularity
of DNNs. Thanks to a generalized nested architecture (which we refer as
\textit{sorted} architecture in this paper) with shared parameters and its
novel update scheme combining random sub-model sampling and a new gradient
accumulation mechanism, SortedNet enables the training of sub-models
simultaneously along with the training of the main model (without any
significant extra training or inference overhead), simplifies dynamic model
selection, customizes deployment during inference, and reduces the model
storage requirement significantly. The versatility and scalability of SortedNet
are validated through various architectures and tasks, including LLaMA, BERT,
RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating
its superiority over existing dynamic training methods. For example, we
introduce a novel adaptive self-speculative approach based on sorted-training
to accelerate large language models decoding. Moreover, SortedNet is able to
train 160 sub-models at once, achieving at least 96\% of the original model's
performance.
