  The transformer neural network has significantly out-shined all other neural
network architectures as the engine behind large language models. We provide a
theoretical analysis of the expressivity of the transformer architecture
through the lens of topos theory. From this viewpoint, we show that many common
neural network architectures, such as the convolutional, recurrent and graph
convolutional networks, can be embedded in a pretopos of piecewise-linear
functions, but that the transformer necessarily lives in its topos completion.
In particular, this suggests that the two network families instantiate
different fragments of logic: the former are first order, whereas transformers
are higher-order reasoners. Furthermore, we draw parallels with architecture
search and gradient descent, integrating our analysis in the framework of
cybernetic agents.
