  Accurately gauging uncertainty on the underlying environment is a
longstanding goal of intelligent systems. We characterize which latent concepts
pre-trained sequence models are naturally able to reason with. We go back to De
Finetti's predictive view of Bayesian reasoning: instead of modeling latent
parameters through priors and likelihoods like topic models do, De Finetti has
long advocated for modeling exchangeable (permutation invariant) sequences of
observables. According to this view, pre-training autoregressive models
formulates informed beliefs based on prior observations ("empirical Bayes"),
and forward generation is a simulated instantiation of an environment
("posterior inference"). This connection allows extending in-context learning
(ICL) beyond predictive settings, highlighting sequence models' ability to
perform explicit statistical inference. In particular, we show the sequence
prediction loss over exchangeable documents controls performance on downstream
tasks where uncertainty quantification is key. Empirically, we propose and
demonstrate several approaches for encoding exchangeability in sequence model
architectures: data augmentation, regularization, and causal masking.
