  Large language models (LLMs) are known for their exceptional performance in
natural language processing, making them highly effective in many human
life-related or even job-related tasks. The attention mechanism in the
Transformer architecture is a critical component of LLMs, as it allows the
model to selectively focus on specific input parts. The softmax unit, which is
a key part of the attention mechanism, normalizes the attention scores. Hence,
the performance of LLMs in various NLP tasks depends significantly on the
crucial role played by the attention mechanism with the softmax unit.
  In-context learning, as one of the celebrated abilities of recent LLMs, is an
important concept in querying LLMs such as ChatGPT. Without further parameter
updates, Transformers can learn to predict based on few in-context examples.
However, the reason why Transformers becomes in-context learners is not well
understood. Recently, several works [ASA+22,GTLV22,ONR+22] have studied the
in-context learning from a mathematical perspective based on a linear
regression formulation $\min_x\| Ax - b \|_2$, which show Transformers'
capability of learning linear functions in context.
  In this work, we study the in-context learning based on a softmax regression
formulation $\min_{x} \| \langle \exp(Ax), {\bf 1}_n \rangle^{-1} \exp(Ax) - b
\|_2$ of Transformer's attention mechanism. We show the upper bounds of the
data transformations induced by a single self-attention layer and by
gradient-descent on a $\ell_2$ regression loss for softmax prediction function,
which imply that when training self-attention-only Transformers for fundamental
regression tasks, the models learned by gradient-descent and Transformers show
great similarity.
