  We study the probabilistic modeling performed by Autoregressive Large
Language Models (LLMs) through the angle of time directionality, addressing a
question first raised in (Shannon, 1951). For large enough models, we
empirically find a time asymmetry in their ability to learn natural language: a
difference in the average log-perplexity when trying to predict the next token
versus when trying to predict the previous one. This difference is at the same
time subtle and very consistent across various modalities (language, model
size, training time, ...). Theoretically, this is surprising: from an
information-theoretic point of view, there should be no such difference. We
provide a theoretical framework to explain how such an asymmetry can appear
from sparsity and computational complexity considerations, and outline a number
of perspectives opened by our results.
