  We prove a universal approximation property (UAP) for a class of ODENet and a
class of ResNet, which are simplified mathematical models for deep learning
systems with skip connections. The UAP can be stated as follows. Let $n$ and
$m$ be the dimension of input and output data, and assume $m\leq n$. Then we
show that ODENet of width $n+m$ with any non-polynomial continuous activation
function can approximate any continuous function on a compact subset on
$\mathbb{R}^n$. We also show that ResNet has the same property as the depth
tends to infinity. Furthermore, we derive the gradient of a loss function
explicitly with respect to a certain tuning variable. We use this to construct
a learning algorithm for ODENet. To demonstrate the usefulness of this
algorithm, we apply it to a regression problem, a binary classification, and a
multinomial classification in MNIST.
