  Large language models (LLMs) are demonstrating remarkable capabilities across
various tasks despite lacking a foundation in human cognition. This raises the
question: can these models, beyond simply mimicking human language patterns,
offer insights into the mechanisms underlying human cognition? This study
explores the ability of ChatGPT to predict human performance in a
language-based memory task. Building upon theories of text comprehension, we
hypothesize that recognizing ambiguous sentences (e.g., "Because Bill drinks
wine is never kept in the house") is facilitated by preceding them with
contextually relevant information. Participants, both human and ChatGPT, were
presented with pairs of sentences. The second sentence was always a garden-path
sentence designed to be inherently ambiguous, while the first sentence either
provided a fitting (e.g., "Bill has chronic alcoholism") or an unfitting
context (e.g., "Bill likes to play golf"). We measured both human's and
ChatGPT's ratings of sentence relatedness, ChatGPT's memorability ratings for
the garden-path sentences, and humans' spontaneous memory for the garden-path
sentences. The results revealed a striking alignment between ChatGPT's
assessments and human performance. Sentences deemed more related and assessed
as being more memorable by ChatGPT were indeed better remembered by humans,
even though ChatGPT's internal mechanisms likely differ significantly from
human cognition. This finding, which was confirmed with a robustness check
employing synonyms, underscores the potential of generative AI models to
predict human performance accurately. We discuss the broader implications of
these findings for leveraging LLMs in the development of psychological theories
and for gaining a deeper understanding of human cognition.
