  In this work we systematically analyze general properties of differential
equations used as machine learning models. We demonstrate that the gradient of
the loss function with respect to to the hidden state can be considered as a
generalized momentum conjugate to the hidden state, allowing application of the
tools of classical mechanics. In addition, we show that not only residual
networks, but also feedforward neural networks with small nonlinearities and
the weights matrices deviating only slightly from identity matrices can be
related to the differential equations. We propose a differential equation
describing such networks and investigate its properties.
