  Understanding the connections between artificial and biological intelligent
systems can reveal fundamental principles underlying general intelligence.
While many artificial intelligence (AI) models have a neuroscience counterpart,
such connections are largely missing in Transformer models and the
self-attention mechanism. Here, we examine the relationship between attention
heads and human episodic memory. We focus on the induction heads, which
contribute to the in-context learning capabilities of Transformer-based large
language models (LLMs). We demonstrate that induction heads are behaviorally,
functionally, and mechanistically similar to the contextual maintenance and
retrieval (CMR) model of human episodic memory. Our analyses of LLMs
pre-trained on extensive text data show that CMR-like heads often emerge in the
intermediate model layers and that their behavior qualitatively mirrors the
memory biases seen in humans. Our findings uncover a parallel between the
computational mechanisms of LLMs and human memory, offering valuable insights
into both research fields.
