  Reasoning is a key ability for an intelligent system. Large language models
(LMs) achieve above-chance performance on abstract reasoning tasks, but exhibit
many imperfections. However, human abstract reasoning is also imperfect. For
example, human reasoning is affected by our real-world knowledge and beliefs,
and shows notable "content effects"; humans reason more reliably when the
semantic content of a problem supports the correct logical inferences. These
content-entangled reasoning patterns play a central role in debates about the
fundamental nature of human intelligence. Here, we investigate whether language
models $\unicode{x2014}$ whose prior expectations capture some aspects of human
knowledge $\unicode{x2014}$ similarly mix content into their answers to logical
problems. We explored this question across three logical reasoning tasks:
natural language inference, judging the logical validity of syllogisms, and the
Wason selection task. We evaluate state of the art large language models, as
well as humans, and find that the language models reflect many of the same
patterns observed in humans across these tasks $\unicode{x2014}$ like humans,
models answer more accurately when the semantic content of a task supports the
logical inferences. These parallels are reflected both in answer patterns, and
in lower-level features like the relationship between model answer
distributions and human response times. Our findings have implications for
understanding both these cognitive effects in humans, and the factors that
contribute to language model performance.
