  To generate coherent responses, language models infer unobserved meaning from
their input text sequence. One potential explanation for this capability arises
from theories of delay embeddings in dynamical systems, which prove that
unobserved variables can be recovered from the history of only a handful of
observed variables. To test whether language models are effectively
constructing delay embeddings, we measure the capacities of sequence models to
reconstruct unobserved dynamics. We trained 1-layer transformer decoders and
state-space sequence models on next-step prediction from noisy,
partially-observed time series data. We found that each sequence layer can
learn a viable embedding of the underlying system. However, state-space models
have a stronger inductive bias than transformers-in particular, they more
effectively reconstruct unobserved information at initialization, leading to
more parameter-efficient models and lower error on dynamics tasks. Our work
thus forges a novel connection between dynamical systems and deep learning
sequence models via delay embedding theory.
