  It has been observed that residual networks can be viewed as the explicit
Euler discretization of an Ordinary Differential Equation (ODE). This
observation motivated the introduction of so-called Neural ODEs, which allow
more general discretization schemes with adaptive time stepping. Here, we
propose ANODEV2, which is an extension of this approach that also allows
evolution of the neural network parameters, in a coupled ODE-based formulation.
The Neural ODE method introduced earlier is in fact a special case of this new
more general framework. We present the formulation of ANODEV2, derive
optimality conditions, and implement a coupled reaction-diffusion-advection
version of this framework in PyTorch. We present empirical results using
several different configurations of ANODEV2, testing them on multiple models on
CIFAR-10. We report results showing that this coupled ODE-based framework is
indeed trainable, and that it achieves higher accuracy, as compared to the
baseline models as well as the recently-proposed Neural ODE approach.
