  We propose to extract meaning representations from autoregressive language
models by considering the distribution of all possible trajectories extending
an input text. This strategy is prompt-free, does not require fine-tuning, and
is applicable to any pre-trained autoregressive model. Moreover, unlike
vector-based representations, distribution-based representations can also model
asymmetric relations (e.g., direction of logical entailment, hypernym/hyponym
relations) by using algebraic operations between likelihood functions. These
ideas are grounded in distributional perspectives on semantics and are
connected to standard constructions in automata theory, but to our knowledge
they have not been applied to modern language models. We empirically show that
the representations obtained from large models align well with human
annotations, outperform other zero-shot and prompt-free methods on semantic
similarity tasks, and can be used to solve more complex entailment and
containment tasks that standard embeddings cannot handle. Finally, we extend
our method to represent data from different modalities (e.g., image and text)
using multimodal autoregressive models. Our code is available at:
https://github.com/tianyu139/meaning-as-trajectories
