  Open-domain question answering (QA) is known to involve several underlying
knowledge and reasoning challenges, but are models actually learning such
knowledge when trained on benchmark tasks? To investigate this, we introduce
several new challenge tasks that probe whether state-of-the-art QA models have
general knowledge about word definitions and general taxonomic reasoning, both
of which are fundamental to more complex forms of reasoning and are widespread
in benchmark datasets. As an alternative to expensive crowd-sourcing, we
introduce a methodology for automatically building datasets from various types
of expert knowledge (e.g., knowledge graphs and lexical taxonomies), allowing
for systematic control over the resulting probes and for a more comprehensive
evaluation. We find automatically constructing probes to be vulnerable to
annotation artifacts, which we carefully control for. Our evaluation confirms
that transformer-based QA models are already predisposed to recognize certain
types of structural lexical knowledge. However, it also reveals a more nuanced
picture: their performance degrades substantially with even a slight increase
in the number of hops in the underlying taxonomic hierarchy, or as more
challenging distractor candidate answers are introduced. Further, even when
these models succeed at the standard instance-level evaluation, they leave much
room for improvement when assessed at the level of clusters of semantically
connected probes (e.g., all Isa questions about a concept).
