  Modeling spatiotemporal dynamical systems is a fundamental challenge in
machine learning. Transformer models have been very successful in NLP and
computer vision where they provide interpretable representations of data.
However, a limitation of transformers in modeling continuous dynamical systems
is that they are fundamentally discrete time and space models and thus have no
guarantees regarding continuous sampling. To address this challenge, we present
the Continuous Spatiotemporal Transformer (CST), a new transformer architecture
that is designed for the modeling of continuous systems. This new framework
guarantees a continuous and smooth output via optimization in Sobolev space. We
benchmark CST against traditional transformers as well as other spatiotemporal
dynamics modeling methods and achieve superior performance in a number of tasks
on synthetic and real systems, including learning brain dynamics from calcium
imaging data.
