  Existing work has analyzed the representational capacity of the transformer
architecture by means of formal models of computation. However, the focus so
far has been on analyzing the architecture in terms of language
\emph{acceptance}. We contend that this is an ill-suited problem in the study
of \emph{language models} (LMs), which are definitionally \emph{probability
distributions} over strings. In this paper, we focus on the relationship
between transformer LMs and $n$-gram LMs, a simple and historically relevant
class of language models. We show that transformer LMs using the hard or sparse
attention mechanisms can exactly represent any $n$-gram LM, giving us a
concrete lower bound on their probabilistic representational capacity. This
provides a first step towards understanding the mechanisms that transformer LMs
can use to represent probability distributions over strings.
