  Neural network language models can serve as computational hypotheses about
how humans process language. We compared the model-human consistency of diverse
language models using a novel experimental approach: controversial sentence
pairs. For each controversial sentence pair, two language models disagree about
which sentence is more likely to occur in natural text. Considering nine
language models (including n-gram, recurrent neural networks, and transformer
models), we created hundreds of such controversial sentence pairs by either
selecting sentences from a corpus or synthetically optimizing sentence pairs to
be highly controversial. Human subjects then provided judgments indicating for
each pair which of the two sentences is more likely. Controversial sentence
pairs proved highly effective at revealing model failures and identifying
models that aligned most closely with human judgments. The most
human-consistent model tested was GPT-2, although experiments also revealed
significant shortcomings of its alignment with human perception.
