  Large language models (LLMs) have numerous real-life applications across
various domains, such as natural language translation, sentiment analysis,
language modeling, chatbots and conversational agents, creative writing, text
classification, summarization, and generation. LLMs have shown great promise in
improving the accuracy and efficiency of these tasks, and have the potential to
revolutionize the field of natural language processing (NLP) in the years to
come.
  Exponential function based attention unit is a fundamental element in LLMs.
Several previous works have studied the convergence of exponential regression
and softmax regression.
  The exponential regression [Li, Song, Zhou 2023] and softmax regression
[Deng, Li, Song 2023] can be formulated as follows. Given matrix $A \in
\mathbb{R}^{n \times d}$ and vector $b \in \mathbb{R}^n$, the goal of
exponential regression is to solve \begin{align*} \min_{x} \| \exp(Ax) - b \|_2
\end{align*} and the goal of softmax regression is to solve \begin{align*}
\min_{x} \| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2 .
\end{align*}
  In this work, we define a slightly different formulation than softmax
regression. \begin{align*} \min_{x \in \mathbb{R}^d } \| u(x) - \langle u(x) ,
{\bf 1}_n \rangle \cdot b \|_2 \end{align*} where $u(x) \in \{ \exp(Ax),
\cosh(Ax) , \sinh(Ax) \}$. We provide an input sparsity time algorithm for this
problem. Our algorithm framework is very general and can be applied to
functions like $\cosh()$ and $\sinh()$ as well. Our technique is also general
enough to be applied to in-context learning for rescaled softmax regression.
