  In psycholinguistics, the creation of controlled materials is crucial to
ensure that research outcomes are solely attributed to the intended
manipulations and not influenced by extraneous factors. To achieve this,
psycholinguists typically pretest linguistic materials, where a common pretest
is to solicit plausibility judgments from human evaluators on specific
sentences. In this work, we investigate whether Language Models (LMs) can be
used to generate these plausibility judgements. We investigate a wide range of
LMs across multiple linguistic structures and evaluate whether their
plausibility judgements correlate with human judgements. We find that GPT-4
plausibility judgements highly correlate with human judgements across the
structures we examine, whereas other LMs correlate well with humans on commonly
used syntactic structures. We then test whether this correlation implies that
LMs can be used instead of humans for pretesting. We find that when
coarse-grained plausibility judgements are needed, this works well, but when
fine-grained judgements are necessary, even GPT-4 does not provide satisfactory
discriminative power.
