  Neural Ordinary Differential Equations (NODEs) are a new class of models that
transform data continuously through infinite-depth architectures. The
continuous nature of NODEs has made them particularly suitable for learning the
dynamics of complex physical systems. While previous work has mostly been
focused on first order ODEs, the dynamics of many systems, especially in
classical physics, are governed by second order laws. In this work, we consider
Second Order Neural ODEs (SONODEs). We show how the adjoint sensitivity method
can be extended to SONODEs and prove that the optimisation of a first order
coupled ODE is equivalent and computationally more efficient. Furthermore, we
extend the theoretical understanding of the broader class of Augmented NODEs
(ANODEs) by showing they can also learn higher order dynamics with a minimal
number of augmented dimensions, but at the cost of interpretability. This
indicates that the advantages of ANODEs go beyond the extra space offered by
the augmented dimensions, as originally thought. Finally, we compare SONODEs
and ANODEs on synthetic and real dynamical systems and demonstrate that the
inductive biases of the former generally result in faster training and better
performance.
