  In recent years, deep learning has been connected with optimal control as a
way to define a notion of a continuous underlying learning problem. In this
view, neural networks can be interpreted as a discretization of a parametric
Ordinary Differential Equation which, in the limit, defines a continuous-depth
neural network. The learning task then consists in finding the best ODE
parameters for the problem under consideration, and their number increases with
the accuracy of the time discretization. Although important steps have been
taken to realize the advantages of such continuous formulations, most current
learning techniques fix a discretization (i.e. the number of layers is fixed).
In this work, we propose an iterative adaptive algorithm where we progressively
refine the time discretization (i.e. we increase the number of layers).
Provided that certain tolerances are met across the iterations, we prove that
the strategy converges to the underlying continuous problem. One salient
advantage of such a shallow-to-deep approach is that it helps to benefit in
practice from the higher approximation properties of deep networks by
mitigating over-parametrization issues. The performance of the approach is
illustrated in several numerical examples.
