  Transformer-based language models are treated as black-boxes because of their
large number of parameters and complex internal interactions, which is a
serious safety concern. Mechanistic Interpretability (MI) intends to
reverse-engineer neural network behaviors in terms of human-understandable
components. In this work, we focus on understanding how GPT-2 Small performs
the task of predicting three-letter acronyms. Previous works in the MI field
have focused so far on tasks that predict a single token. To the best of our
knowledge, this is the first work that tries to mechanistically understand a
behavior involving the prediction of multiple consecutive tokens. We discover
that the prediction is performed by a circuit composed of 8 attention heads
(~5% of the total heads) which we classified in three groups according to their
role. We also demonstrate that these heads concentrate the acronym prediction
functionality. In addition, we mechanistically interpret the most relevant
heads of the circuit and find out that they use positional information which is
propagated via the causal mask mechanism. We expect this work to lay the
foundation for understanding more complex behaviors involving multiple-token
predictions.
