  Are the predictions of humans and language models affected by similar things?
Research suggests that while comprehending language, humans make predictions
about upcoming words, with more predictable words being processed more easily.
However, evidence also shows that humans display a similar processing advantage
for highly anomalous words when these words are semantically related to the
preceding context or to the most probable continuation. Using stimuli from 3
psycholinguistic experiments, we find that this is also almost always also the
case for 8 contemporary transformer language models (BERT, ALBERT, RoBERTa,
XLM-R, GPT-2, GPT-Neo, GPT-J, and XGLM). We then discuss the implications of
this phenomenon for our understanding of both human language comprehension and
the predictions made by language models.
