  Transformers are neural networks that revolutionized natural language
processing and machine learning. They process sequences of inputs, like words,
using a mechanism called self-attention, which is trained via masked language
modeling (MLM). In MLM, a word is randomly masked in an input sequence, and the
network is trained to predict the missing word. Despite the practical success
of transformers, it remains unclear what type of data distribution
self-attention can learn efficiently. Here, we show analytically that if one
decouples the treatment of word positions and embeddings, a single layer of
self-attention learns the conditionals of a generalized Potts model with
interactions between sites and Potts colors. Moreover, we show that training
this neural network is exactly equivalent to solving the inverse Potts problem
by the so-called pseudo-likelihood method, well known in statistical physics.
Using this mapping, we compute the generalization error of self-attention in a
model scenario analytically using the replica method.
