  Democratization of machine learning requires architectures that automatically
adapt to new problems. Neural Differential Equations (NDEs) have emerged as a
popular modeling framework by removing the need for ML practitioners to choose
the number of layers in a recurrent model. While we can control the
computational cost by choosing the number of layers in standard architectures,
in NDEs the number of neural network evaluations for a forward pass can depend
on the number of steps of the adaptive ODE solver. But, can we force the NDE to
learn the version with the least steps while not increasing the training cost?
Current strategies to overcome slow prediction require high order automatic
differentiation, leading to significantly higher training time. We describe a
novel regularization method that uses the internal cost heuristics of adaptive
differential equation solvers combined with discrete adjoint sensitivities to
guide the training process towards learning NDEs that are easier to solve. This
approach opens up the blackbox numerical analysis behind the differential
equation solver's algorithm and directly uses its local error estimates and
stiffness heuristics as cheap and accurate cost estimates. We incorporate our
method without any change in the underlying NDE framework and show that our
method extends beyond Ordinary Differential Equations to accommodate Neural
Stochastic Differential Equations. We demonstrate how our approach can halve
the prediction time and, unlike other methods which can increase the training
time by an order of magnitude, we demonstrate similar reduction in training
times. Together this showcases how the knowledge embedded within
state-of-the-art equation solvers can be used to enhance machine learning.
