  We investigate the expressive power of deep residual neural networks
idealized as continuous dynamical systems through control theory. Specifically,
we consider two properties that arise from supervised learning, namely
universal interpolation - the ability to match arbitrary input and target
training samples - and the closely related notion of universal approximation -
the ability to approximate input-target functional relationships via flow maps.
Under the assumption of affine invariance of the control family, we give a
characterisation of universal interpolation, showing that it holds for
essentially any architecture with non-linearity. Furthermore, we elucidate the
relationship between universal interpolation and universal approximation in the
context of general control systems, showing that the two properties cannot be
deduced from each other. At the same time, we identify conditions on the
control family and the target function that ensures the equivalence of the two
notions.
