  We introduce network with sub-networks, a neural network which its weight
layers could be detached into sub-neural networks during inference. To develop
weights and biases which could be inserted in both base and sub-neural
networks, firstly, the parameters are copied from sub-model to base-model. Each
model is forward-propagated separately. Gradients from a pair of networks are
averaged and, used to update both networks. Our base model achieves the
test-accuracy which is comparable to the regularly trained models, while the
model maintains the ability to detach weight layers.
