  Today's best language models still struggle with hallucinations: factually
incorrect generations, which impede their ability to reliably retrieve
information seen during training. The reversal curse, where models cannot
recall information when probed in a different order than was encountered during
training, exemplifies this in information retrieval. We reframe the reversal
curse as a factorization curse - a failure of models to learn the same joint
distribution under different factorizations. Through a series of controlled
experiments with increasing levels of realism including WikiReversal, a setting
we introduce to closely simulate a knowledge intensive finetuning task, we find
that the factorization curse is an inherent failure of the next-token
prediction objective used in popular large language models. Moreover, we
demonstrate reliable information retrieval cannot be solved with scale,
reversed tokens, or even naive bidirectional-attention training. Consequently,
various approaches to finetuning on specialized data would necessarily provide
mixed results on downstream tasks, unless the model has already seen the right
sequence of tokens. Across five tasks of varying levels of complexity, our
results uncover a promising path forward: factorization-agnostic objectives can
significantly mitigate the reversal curse and hint at improved knowledge
storage and planning capabilities.
