  Language models have the ability to perform in-context learning (ICL),
allowing them to flexibly adapt their behavior based on context. This contrasts
with in-weights learning, where information is statically encoded in model
parameters from iterated observations of the data. Despite this apparent
ability to learn in-context, language models are known to struggle when faced
with unseen or rarely seen tokens. Hence, we study $\textbf{structural
in-context learning}$, which we define as the ability of a model to execute
in-context learning on arbitrary tokens -- so called because the model must
generalize on the basis of e.g. sentence structure or task structure, rather
than semantic content encoded in token embeddings. An ideal model would be able
to do both: flexibly deploy in-weights operations (in order to robustly
accommodate ambiguous or unknown contexts using encoded semantic information)
and structural in-context operations (in order to accommodate novel tokens). We
study structural in-context algorithms in a simple part-of-speech setting using
both practical and toy models. We find that active forgetting, a technique that
was recently introduced to help models generalize to new languages, forces
models to adopt structural in-context learning solutions. Finally, we introduce
$\textbf{temporary forgetting}$, a straightforward extension of active
forgetting that enables one to control how much a model relies on in-weights
vs. in-context solutions. Importantly, temporary forgetting allows us to induce
a $\textit{dual process strategy}$ where in-context and in-weights solutions
coexist within a single model.
