  Although it is known that transformer language models (LMs) pass features
from early layers to later layers, it is not well understood how this
information is represented and routed by the model. By analyzing particular
mechanism LMs use to accomplish this, we find that it is also used to recall
items from a list, and show that this mechanism can explain an otherwise
arbitrary-seeming sensitivity of the model to the order of items in the prompt.
Specifically, we find that models write into low-rank subspaces of the residual
stream to represent features which are then read out by specific later layers,
forming low-rank communication channels between layers. By decomposing
attention head weight matrices with the Singular Value Decomposition (SVD), we
find that previously described interactions between heads separated by one or
more layers can be predicted via analysis of their weight matrices. We show
that it is possible to manipulate the internal model representations as well as
edit model weights based on the mechanism we discover in order to significantly
improve performance on our synthetic Laundry List task, which requires recall
from a list, often improving task accuracy by over 20%. Our analysis reveals a
surprisingly intricate interpretable structure learned from language model
pretraining, and helps us understand why sophisticated LMs sometimes fail in
simple domains, facilitating future analysis of more complex behaviors.
