  Large language models (LLMs) are currently at the forefront of intertwining
AI systems with human communication and everyday life. Therefore, it is of
great importance to evaluate their emerging abilities. In this study, we show
that LLMs like GPT-3 exhibit behavior that strikingly resembles human-like
intuition - and the cognitive errors that come with it. However, LLMs with
higher cognitive capabilities, in particular ChatGPT and GPT-4, learned to
avoid succumbing to these errors and perform in a hyperrational manner. For our
experiments, we probe LLMs with the Cognitive Reflection Test (CRT) as well as
semantic illusions that were originally designed to investigate intuitive
decision-making in humans. Our study demonstrates that investigating LLMs with
methods from psychology has the potential to reveal otherwise unknown emergent
traits.
