  Do machines and humans process language in similar ways? Recent research has
hinted in the affirmative, finding that brain signals can be effectively
predicted using the internal representations of language models (LMs). Although
such results are thought to reflect shared computational principles between LMs
and human brains, there are also clear differences in how LMs and humans
represent and use language. In this work, we systematically explore the
divergences between human and machine language processing by examining the
differences between LM representations and human brain responses to language as
measured by Magnetoencephalography (MEG) across two datasets in which subjects
read and listened to narrative stories. Using a data-driven approach, we
identify two domains that are not captured well by LMs: social/emotional
intelligence and physical commonsense. We then validate these domains with
human behavioral experiments and show that fine-tuning LMs on these domains can
improve their alignment with human brain responses.
