  This article investigates the effect of explicitly adding auxiliary algebraic
trajectory information to neural networks for dynamical systems. We draw
inspiration from the field of differential-algebraic equations and differential
equations on manifolds and implement related methods in residual neural
networks, despite some fundamental scenario differences. Constraint or
auxiliary information effects are incorporated through stabilization as well as
projection methods, and we show when to use which method based on experiments
involving simulations of multi-body pendulums and molecular dynamics scenarios.
Several of our methods are easy to implement in existing code and have limited
impact on training performance while giving significant boosts in terms of
inference.
