  Neural operator learning models have emerged as very effective surrogates in
data-driven methods for partial differential equations (PDEs) across different
applications from computational science and engineering. Such operator learning
models not only predict particular instances of a physical or biological system
in real-time but also forecast classes of solutions corresponding to a
distribution of initial and boundary conditions or forcing terms. % DeepONet is
the first neural operator model and has been tested extensively for a broad
class of solutions, including Riemann problems. Transformers have not been used
in that capacity, and specifically, they have not been tested for solutions of
PDEs with low regularity. %
  In this work, we first establish the theoretical groundwork that transformers
possess the universal approximation property as operator learning models.
  We then apply transformers to forecast solutions of diverse dynamical systems
with solutions of finite regularity for a plurality of initial conditions and
forcing terms. In particular, we consider three examples: the Izhikevich neuron
model, the tempered fractional-order Leaky Integrate-and-Fire (LIF) model, and
the one-dimensional Euler equation Riemann problem. For the latter problem, we
also compare with variants of DeepONet, and we find that transformers
outperform DeepONet in accuracy but they are computationally more expensive.
