  Large language models (LLMs) have demonstrated exceptional performance across
various linguistic tasks. However, it remains uncertain whether LLMs have
developed human-like fine-grained grammatical intuition. This preregistered
study (https://osf.io/t5nes) presents the first large-scale investigation of
ChatGPT's grammatical intuition, building upon a previous study that collected
laypeople's grammatical judgments on 148 linguistic phenomena that linguists
judged to be grammatical, ungrammatical, or marginally grammatical (Sprouse,
Schutze, & Almeida, 2013). Our primary focus was to compare ChatGPT with both
laypeople and linguists in the judgement of these linguistic constructions. In
Experiment 1, ChatGPT assigned ratings to sentences based on a given reference
sentence. Experiment 2 involved rating sentences on a 7-point scale, and
Experiment 3 asked ChatGPT to choose the more grammatical sentence from a pair.
Overall, our findings demonstrate convergence rates ranging from 73% to 95%
between ChatGPT and linguists, with an overall point-estimate of 89%.
Significant correlations were also found between ChatGPT and laypeople across
all tasks, though the correlation strength varied by task. We attribute these
results to the psychometric nature of the judgment tasks and the differences in
language processing styles between humans and LLMs.
