  Continuous-depth neural networks can be viewed as deep limits of discrete
neural networks whose dynamics resemble a discretization of an ordinary
differential equation (ODE). Although important steps have been taken to
realize the advantages of such continuous formulations, most current techniques
are not truly continuous-depth as they assume \textit{identical} layers.
Indeed, existing works throw into relief the myriad difficulties presented by
an infinite-dimensional parameter space in learning a continuous-depth neural
ODE. To this end, we introduce a shooting formulation which shifts the
perspective from parameterizing a network layer-by-layer to parameterizing over
optimal networks described only by a set of initial conditions. For
scalability, we propose a novel particle-ensemble parametrization which fully
specifies the optimal weight trajectory of the continuous-depth neural network.
Our experiments show that our particle-ensemble shooting formulation can
achieve competitive performance, especially on long-range forecasting tasks.
Finally, though the current work is inspired by continuous-depth neural
networks, the particle-ensemble shooting formulation also applies to
discrete-time networks and may lead to a new fertile area of research in deep
learning parametrization.
