  This case study investigates the extent to which a language model (GPT-2) is
able to capture native speakers' intuitions about implicit causality in a
sentence completion task. We first reproduce earlier results (showing lower
surprisal values for pronouns that are congruent with either the subject or
object, depending on which one corresponds to the implicit causality bias of
the verb), and then examine the effects of gender and verb frequency on model
performance. Our second study examines the reasoning ability of GPT-2: is the
model able to produce more sensible motivations for why the subject VERBed the
object if the verbs have stronger causality biases? We also developed a
methodology to avoid human raters being biased by obscenities and disfluencies
generated by the model.
