  In-Context Learning (ICL) has been a powerful emergent property of large
language models that has attracted increasing attention in recent years. In
contrast to regular gradient-based learning, ICL is highly interpretable and
does not require parameter updates. In this paper, we show that, for linearized
transformer networks, ICL can be made explicit and permanent through the
inclusion of bias terms. We mathematically demonstrate the equivalence between
a model with ICL demonstration prompts and the same model with the additional
bias terms. Our algorithm (ICLCA) allows for exact conversion in an inexpensive
manner. Existing methods are not exact and require expensive parameter updates.
We demonstrate the efficacy of our approach through experiments that show the
exact incorporation of ICL tokens into a linear transformer. We further suggest
how our method can be adapted to achieve cheap approximate conversion of ICL
tokens, even in regular transformer networks that are not linearized. Our
experiments on GPT-2 show that, even though the conversion is only approximate,
the model still gains valuable context from the included bias terms.
