  Large language models (LLMs) have shown the emergent capability of in-context
learning (ICL). One line of research has explained ICL as functionally
performing gradient descent. In this paper, we introduce a new way of
diagnosing whether ICL is functionally equivalent to gradient-based learning.
Our approach is based on the inverse frequency effect (IFE) -- a phenomenon in
which an error-driven learner is expected to show larger updates when trained
on infrequent examples than frequent ones. The IFE has previously been studied
in psycholinguistics because humans show this effect in the context of
structural priming (the tendency for people to produce sentence structures they
have encountered recently); the IFE has been used as evidence that human
structural priming must involve error-driven learning mechanisms. In our
experiments, we simulated structural priming within ICL and found that LLMs
display the IFE, with the effect being stronger in larger models. We conclude
that ICL is indeed a type of gradient-based learning, supporting the hypothesis
that a gradient component is implicitly computed in the forward pass during
ICL. Our results suggest that both humans and LLMs make use of gradient-based,
error-driven processing mechanisms.
