  In our work, we bridge deep neural network design with numerical differential
equations. We show that many effective networks, such as ResNet, PolyNet,
FractalNet and RevNet, can be interpreted as different numerical
discretizations of differential equations. This finding brings us a brand new
perspective on the design of effective deep architectures. We can take
advantage of the rich knowledge in numerical analysis to guide us in designing
new and potentially more effective deep networks. As an example, we propose a
linear multi-step architecture (LM-architecture) which is inspired by the
linear multi-step method solving ordinary differential equations. The
LM-architecture is an effective structure that can be used on any ResNet-like
networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the
networks obtained by applying the LM-architecture on ResNet and ResNeXt
respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on
both CIFAR and ImageNet with comparable numbers of trainable parameters. In
particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly
compress ($>50$\%) the original networks while maintaining a similar
performance. This can be explained mathematically using the concept of modified
equation from numerical analysis. Last but not least, we also establish a
connection between stochastic control and noise injection in the training
process which helps to improve generalization of the networks. Furthermore, by
relating stochastic training strategy with stochastic dynamic system, we can
easily apply stochastic training to the networks with the LM-architecture. As
an example, we introduced stochastic depth to LM-ResNet and achieve significant
improvement over the original LM-ResNet on CIFAR10.
