  Many deep neural network architectures loosely based on brain networks have
recently been shown to replicate neural firing patterns observed in the brain.
One of the most exciting and promising novel architectures, the Transformer
neural network, was developed without the brain in mind. In this work, we show
that transformers, when equipped with recurrent position encodings, replicate
the precisely tuned spatial representations of the hippocampal formation; most
notably place and grid cells. Furthermore, we show that this result is no
surprise since it is closely related to current hippocampal models from
neuroscience. We additionally show the transformer version offers dramatic
performance gains over the neuroscience version. This work continues to bind
computations of artificial and brain networks, offers a novel understanding of
the hippocampal-cortical interaction, and suggests how wider cortical areas may
perform complex tasks beyond current neuroscience models such as language
comprehension.
