  Prompting is now a dominant method for evaluating the linguistic knowledge of
large language models (LLMs). While other methods directly read out models'
probability distributions over strings, prompting requires models to access
this internal information by processing linguistic input, thereby implicitly
testing a new type of emergent ability: metalinguistic judgment. In this study,
we compare metalinguistic prompting and direct probability measurements as ways
of measuring models' linguistic knowledge. Broadly, we find that LLMs'
metalinguistic judgments are inferior to quantities directly derived from
representations. Furthermore, consistency gets worse as the prompt query
diverges from direct measurements of next-word probabilities. Our findings
suggest that negative results relying on metalinguistic prompts cannot be taken
as conclusive evidence that an LLM lacks a particular linguistic
generalization. Our results also highlight the value that is lost with the move
to closed APIs where access to probability distributions is limited.
