  Chain-of-Thought (CoT) reasoning is known to improve Large Language Models
both empirically and in terms of theoretical approximation power. However, our
understanding of the inner workings and conditions of apparition of CoT
capabilities remains limited. This paper helps fill this gap by demonstrating
how CoT reasoning emerges in transformers in a controlled and interpretable
setting. In particular, we observe the appearance of a specialized attention
mechanism dedicated to iterative reasoning, which we coined "iteration heads".
We track both the emergence and the precise working of these iteration heads
down to the attention level, and measure the transferability of the CoT skills
to which they give rise between tasks.
