  Large language models (LLMs) can produce long, coherent passages of text,
suggesting that LLMs, although trained on next-word prediction, must represent
the latent structure that characterizes a document. Prior work has found that
internal representations of LLMs encode one aspect of latent structure, namely
syntax; here we investigate a complementary aspect, namely the document's topic
structure. We motivate the hypothesis that LLMs capture topic structure by
connecting LLM optimization to implicit Bayesian inference. De Finetti's
theorem shows that exchangeable probability distributions can be represented as
a mixture with respect to a latent generating distribution. Although text is
not exchangeable at the level of syntax, exchangeability is a reasonable
starting assumption for topic structure. We thus hypothesize that predicting
the next token in text will lead LLMs to recover latent topic distributions. We
examine this hypothesis using Latent Dirichlet Allocation (LDA), an
exchangeable probabilistic topic model, as a target, and we show that the
representations formed by LLMs encode both the topics used to generate
synthetic data and those used to explain natural corpus data.
