  Complex systems are often decomposed into modular subsystems for engineering
tractability. Although various equation based white-box modeling techniques
make use of such structure, learning based methods have yet to incorporate
these ideas broadly. We present a modular simulation framework for modeling
homogeneous multibody dynamical systems, which combines ideas from graph neural
networks and neural differential equations. We learn to model the individual
dynamical subsystem as a neural ODE module. Full simulation of the composite
system is orchestrated via spatio-temporal message passing between these
modules. An arbitrary number of modules can be combined to simulate systems of
a wide variety of coupling topologies. We evaluate our framework on a variety
of systems and show that message passing allows coordination between multiple
modules over time for accurate predictions and in certain cases, enables
zero-shot generalization to new system configurations. Furthermore, we show
that our models can be transferred to new system configurations with lower data
requirement and training effort, compared to those trained from scratch.
