  Transformers, and the attention mechanism in particular, have become
ubiquitous in machine learning. Their success in modeling nonlocal, long-range
correlations has led to their widespread adoption in natural language
processing, computer vision, and time-series problems. Neural operators, which
map spaces of functions into spaces of functions, are necessarily both
nonlinear and nonlocal if they are universal; it is thus natural to ask whether
the attention mechanism can be used in the design of neural operators.
Motivated by this, we study transformers in the function space setting. We
formulate attention as a map between infinite dimensional function spaces and
prove that the attention mechanism as implemented in practice is a Monte Carlo
or finite difference approximation of this operator. The function space
formulation allows for the design of transformer neural operators, a class of
architectures designed to learn mappings between function spaces, for which we
prove a universal approximation result. The prohibitive cost of applying the
attention operator to functions defined on multi-dimensional domains leads to
the need for more efficient attention-based architectures. For this reason we
also introduce a function space generalization of the patching strategy from
computer vision, and introduce a class of associated neural operators.
Numerical results, on an array of operator learning problems, demonstrate the
promise of our approaches to function space formulations of attention and their
use in neural operators.
