  Pre-training by language modeling has become a popular and successful
approach to NLP tasks, but we have yet to understand exactly what linguistic
capacities these pre-training processes confer upon models. In this paper we
introduce a suite of diagnostics drawn from human language experiments, which
allow us to ask targeted questions about the information used by language
models for generating predictions in context. As a case study, we apply these
diagnostics to the popular BERT model, finding that it can generally
distinguish good from bad completions involving shared category or role
reversal, albeit with less sensitivity than humans, and it robustly retrieves
noun hypernyms, but it struggles with challenging inferences and role-based
event prediction -- and in particular, it shows clear insensitivity to the
contextual impacts of negation.
