  This paper presents Mechanistic Neural Networks, a neural network design for
machine learning applications in the sciences. It incorporates a new
Mechanistic Block in standard architectures to explicitly learn governing
differential equations as representations, revealing the underlying dynamics of
data and enhancing interpretability and efficiency in data modeling. Central to
our approach is a novel Relaxed Linear Programming Solver (NeuRLP) inspired by
a technique that reduces solving linear ODEs to solving linear programs. This
integrates well with neural networks and surpasses the limitations of
traditional ODE solvers enabling scalable GPU parallel processing. Overall,
Mechanistic Neural Networks demonstrate their versatility for scientific
machine learning applications, adeptly managing tasks from equation discovery
to dynamic systems modeling. We prove their comprehensive capabilities in
analyzing and interpreting complex scientific data across various applications,
showing significant performance against specialized state-of-the-art methods.
