  Conditional computation and modular networks have been recently proposed for
multitask learning and other problems as a way to decompose problem solving
into multiple reusable computational blocks. We propose a new approach for
learning modular networks based on the isometric version of ResNet with all
residual blocks having the same configuration and the same number of
parameters. This architectural choice allows adding, removing and changing the
order of residual blocks. In our method, the modules can be invoked repeatedly
and allow knowledge transfer to novel tasks by adjusting the order of
computation. This allows soft weight sharing between tasks with only a small
increase in the number of parameters. We show that our method leads to
interpretable self-organization of modules in case of multi-task learning,
transfer learning and domain adaptation while achieving competitive results on
those tasks. From practical perspective, our approach allows to: (a) reuse
existing modules for learning new task by adjusting the computation order, (b)
use it for unsupervised multi-source domain adaptation to illustrate that
adaptation to unseen data can be achieved by only manipulating the order of
pretrained modules, (c) show how our approach can be used to increase accuracy
of existing architectures for image classification tasks such as ImageNet,
without any parameter increase, by reusing the same block multiple times.
