  Training neural networks sequentially in time to approximate solution fields
of time-dependent partial differential equations can be beneficial for
preserving causality and other physics properties; however, the
sequential-in-time training is numerically challenging because training errors
quickly accumulate and amplify over time. This work introduces Neural Galerkin
schemes that update randomized sparse subsets of network parameters at each
time step. The randomization avoids overfitting locally in time and so helps
prevent the error from accumulating quickly over the sequential-in-time
training, which is motivated by dropout that addresses a similar issue of
overfitting due to neuron co-adaptation. The sparsity of the update reduces the
computational costs of training without losing expressiveness because many of
the network parameters are redundant locally at each time step. In numerical
experiments with a wide range of evolution equations, the proposed scheme with
randomized sparse updates is up to two orders of magnitude more accurate at a
fixed computational budget and up to two orders of magnitude faster at a fixed
accuracy than schemes with dense updates.
