  Using vanilla NeuralODEs to model large and/or complex systems often fails
due two reasons: Stability and convergence. NeuralODEs are capable of
describing stable as well as instable dynamic systems. Selecting an appropriate
numerical solver is not trivial, because NeuralODE properties change during
training. If the NeuralODE becomes more stiff, a suboptimal solver may need to
perform very small solver steps, which significantly slows down the training
process. If the NeuralODE becomes to instable, the numerical solver might not
be able to solve it at all, which causes the training process to terminate.
Often, this is tackled by choosing a computational expensive solver that is
robust to instable and stiff ODEs, but at the cost of a significantly decreased
training performance. Our method on the other hand, allows to enforce ODE
properties that fit a specific solver or application-related boundary
conditions. Concerning the convergence behavior, NeuralODEs often tend to run
into local minima, especially if the system to be learned is highly dynamic
and/or oscillating over multiple periods. Because of the vanishing gradient at
a local minimum, the NeuralODE is often not capable of leaving it and converge
to the right solution. We present a technique to add knowledge of ODE
properties based on eigenvalues - like (partly) stability, oscillation
capability, frequency, damping and/or stiffness - to the training objective of
a NeuralODE. We exemplify our method at a linear as well as a nonlinear system
model and show, that the presented training process is far more robust against
local minima, instabilities and sparse data samples and improves training
convergence and performance.
