  When several limited power devices are available, one of the most efficient
ways to make profit of these resources, while reducing the processing latency
and communication load, is to run in parallel several neural sub-networks and
to fuse the result at the end of processing. However, such a combination of
sub-networks must be trained specifically for each particular configuration of
devices (characterized by number of devices and their capacities) which may
vary over different model deployments and even within the same deployment. In
this work we introduce parallelly distributable slimmable (ParaDiS) neural
networks that are splittable in parallel among various device configurations
without retraining. While inspired by slimmable networks allowing instant
adaptation to resources on just one device, ParaDiS networks consist of several
multi-device distributable configurations or switches that strongly share the
parameters between them. We evaluate ParaDiS framework on MobileNet v1 and
ResNet-50 architectures on ImageNet classification task and WDSR architecture
for image super-resolution task. We show that ParaDiS switches achieve similar
or better accuracy than the individual models, i.e., distributed models of the
same structure trained individually. Moreover, we show that, as compared to
universally slimmable networks that are not distributable, the accuracy of
distributable ParaDiS switches either does not drop at all or drops by a
maximum of 1 % only in the worst cases. Finally, once distributed over several
devices, ParaDiS outperforms greatly slimmable models.
