  Neural solvers for partial differential equations (PDEs) have great potential
to generate fast and accurate physics solutions, yet their practicality is
currently limited by their generalizability. PDEs evolve over broad scales and
exhibit diverse behaviors; predicting these phenomena will require learning
representations across a wide variety of inputs which may encompass different
coefficients, boundary conditions, resolutions, or even equations. As a step
towards generalizable PDE modeling, we adapt masked pretraining for physics
problems. Through self-supervised learning across PDEs, masked autoencoders can
consolidate heterogeneous physics to learn meaningful latent representations
and perform latent PDE arithmetic in this space. Furthermore, we demonstrate
that masked pretraining can improve PDE coefficient regression and the
classification of PDE features. Lastly, conditioning neural solvers on learned
latent representations can improve time-stepping and super-resolution
performance across a variety of coefficients, discretizations, or boundary
conditions, as well as on unseen PDEs. We hope that masked pretraining can
emerge as a unifying method across large, unlabeled, and heterogeneous datasets
to learn latent physics at scale.
