  Transformer-based language models (LMs) continue to advance state-of-the-art
performance on NLP benchmark tasks, including tasks designed to mimic
human-inspired "commonsense" competencies. To better understand the degree to
which LMs can be said to have certain linguistic reasoning skills, researchers
are beginning to adapt the tools and concepts of the field of psychometrics.
But to what extent can the benefits flow in the other direction? I.e., can LMs
be of use in predicting what the psychometric properties of test items will be
when those items are given to human participants? We gather responses from
numerous human participants and LMs (transformer and non-transformer-based) on
a broad diagnostic test of linguistic competencies. We then use the responses
to calculate standard psychometric properties of the items in the diagnostic
test, using the human responses and the LM responses separately. We then
determine how well these two sets of predictions match. We find cases in which
transformer-based LMs predict psychometric properties consistently well in
certain categories but consistently poorly in others, thus providing new
insights into fundamental similarities and differences between human and LM
reasoning.
