  Large Language Models (LLMs) have come closest among all models to date to
mastering human language, yet opinions about their linguistic and cognitive
capabilities remain split. Here, we evaluate LLMs using a distinction between
formal linguistic competence -- knowledge of linguistic rules and patterns --
and functional linguistic competence -- understanding and using language in the
world. We ground this distinction in human neuroscience, which has shown that
formal and functional competence rely on different neural mechanisms. Although
LLMs are surprisingly good at formal competence, their performance on
functional competence tasks remains spotty and often requires specialized
fine-tuning and/or coupling with external modules. We posit that models that
use language in human-like ways would need to master both of these competence
types, which, in turn, could require the emergence of mechanisms specialized
for formal linguistic competence, distinct from functional competence.
