  Solving Partial Differential Equations (PDEs) is the core of many fields of
science and engineering. While classical approaches are often prohibitively
slow, machine learning models often fail to incorporate complete system
information. Over the past few years, transformers have had a significant
impact on the field of Artificial Intelligence and have seen increased usage in
PDE applications. However, despite their success, transformers currently lack
integration with physics and reasoning. This study aims to address this issue
by introducing PITT: Physics Informed Token Transformer. The purpose of PITT is
to incorporate the knowledge of physics by embedding partial differential
equations (PDEs) into the learning process. PITT uses an equation tokenization
method to learn an analytically-driven numerical update operator. By tokenizing
PDEs and embedding partial derivatives, the transformer models become aware of
the underlying knowledge behind physical processes. To demonstrate this, PITT
is tested on challenging 1D and 2D PDE neural operator prediction tasks. The
results show that PITT outperforms popular neural operator models and has the
ability to extract physically relevant information from governing equations.
