  By positing a relationship between naturalistic reading times and
information-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008)
provides a natural interface between language models and psycholinguistic
models. This paper re-evaluates a claim due to Goodkind and Bicknell (2018)
that a language model's ability to model reading times is a linear function of
its perplexity. By extending Goodkind and Bicknell's analysis to modern neural
architectures, we show that the proposed relation does not always hold for Long
Short-Term Memory networks, Transformers, and pre-trained models. We introduce
an alternate measure of language modeling performance called predictability
norm correlation based on Cloze probabilities measured from human subjects. Our
new metric yields a more robust relationship between language model quality and
psycholinguistic modeling performance that allows for comparison between models
with different training configurations.
