  Residual networks (ResNets) are a deep learning architecture that
substantially improved the state of the art performance in certain supervised
learning tasks. Since then, they have received continuously growing attention.
ResNets have a recursive structure $x_{k+1} = x_k + R_k(x_k)$ where $R_k$ is a
neural network called a residual block. This structure can be seen as the Euler
discretisation of an associated ordinary differential equation (ODE) which is
called a neural ODE. Recently, ResNets were proposed as the space-time
approximation of ODEs which are not of this neural type. To elaborate this
connection we show that by increasing the number of residual blocks as well as
their expressivity the solution of an arbitrary ODE can be approximated in
space and time simultaneously by deep ReLU ResNets. Further, we derive
estimates on the complexity of the residual blocks required to obtain a
prescribed accuracy under certain regularity assumptions.
