  Transformers have become pivotal in Natural Language Processing,
demonstrating remarkable success in applications like Machine Translation and
Summarization. Given their widespread adoption, several works have attempted to
analyze the expressivity of Transformers. Expressivity of a neural network is
the class of functions it can approximate. A neural network is fully expressive
if it can act as a universal function approximator. We attempt to analyze the
same for Transformers. Contrary to existing claims, our findings reveal that
Transformers struggle to reliably approximate smooth functions, relying on
piecewise constant approximations with sizable intervals. The central question
emerges as: "Are Transformers truly Universal Function Approximators?" To
address this, we conduct a thorough investigation, providing theoretical
insights and supporting evidence through experiments. Theoretically, we prove
that Transformer Encoders cannot approximate smooth functions. Experimentally,
we complement our theory and show that the full Transformer architecture cannot
approximate smooth functions. By shedding light on these challenges, we
advocate a refined understanding of Transformers' capabilities.
