  We compare the discretize-optimize (Disc-Opt) and optimize-discretize
(Opt-Disc) approaches for time-series regression and continuous normalizing
flows (CNFs) using neural ODEs. Neural ODEs are ordinary differential equations
(ODEs) with neural network components. Training a neural ODE is an optimal
control problem where the weights are the controls and the hidden features are
the states. Every training iteration involves solving an ODE forward and
another backward in time, which can require large amounts of computation, time,
and memory. Comparing the Opt-Disc and Disc-Opt approaches in image
classification tasks, Gholami et al. (2019) suggest that Disc-Opt is preferable
due to the guaranteed accuracy of gradients. In this paper, we extend the
comparison to neural ODEs for time-series regression and CNFs. Unlike in
classification, meaningful models in these tasks must also satisfy additional
requirements beyond accurate final-time output, e.g., the invertibility of the
CNF. Through our numerical experiments, we demonstrate that with careful
numerical treatment, Disc-Opt methods can achieve similar performance as
Opt-Disc at inference with drastically reduced training costs. Disc-Opt reduced
costs in six out of seven separate problems with training time reduction
ranging from 39% to 97%, and in one case, Disc-Opt reduced training from nine
days to less than one day.
