  Neural sequence models, especially transformers, exhibit a remarkable
capacity for in-context learning. They can construct new predictors from
sequences of labeled examples $(x, f(x))$ presented in the input without
further parameter updates. We investigate the hypothesis that transformer-based
in-context learners implement standard learning algorithms implicitly, by
encoding smaller models in their activations, and updating these implicit
models as new examples appear in the context. Using linear regression as a
prototypical problem, we offer three sources of evidence for this hypothesis.
First, we prove by construction that transformers can implement learning
algorithms for linear models based on gradient descent and closed-form ridge
regression. Second, we show that trained in-context learners closely match the
predictors computed by gradient descent, ridge regression, and exact
least-squares regression, transitioning between different predictors as
transformer depth and dataset noise vary, and converging to Bayesian estimators
for large widths and depths. Third, we present preliminary evidence that
in-context learners share algorithmic features with these predictors: learners'
late layers non-linearly encode weight vectors and moment matrices. These
results suggest that in-context learning is understandable in algorithmic
terms, and that (at least in the linear case) learners may rediscover standard
estimation algorithms. Code and reference implementations are released at
https://github.com/ekinakyurek/google-research/blob/master/incontext.
