  Developing fast surrogates for Partial Differential Equations (PDEs) will
accelerate design and optimization in almost all scientific and engineering
applications. Neural networks have been receiving ever-increasing attention and
demonstrated remarkable success in computational modeling of PDEs, however;
their prediction accuracy is not at the level of full deployment. In this work,
we utilize the transformer architecture, the backbone of numerous
state-of-the-art AI models, to learn the dynamics of physical systems as the
mixing of spatial patterns learned by a convolutional autoencoder. Moreover, we
incorporate the idea of multi-scale hierarchical time-stepping to increase the
prediction speed and decrease accumulated error over time. Our model achieves
similar or better results in predicting the time-evolution of Navier-Stokes
equations compared to the powerful Fourier Neural Operator (FNO) and two
transformer-based neural operators OFormer and Galerkin Transformer.
