  Most benchmark datasets targeting commonsense reasoning focus on everyday
scenarios: physical knowledge like knowing that you could fill a cup under a
waterfall [Talmor et al., 2019], social knowledge like bumping into someone is
awkward [Sap et al., 2019], and other generic situations. However, there is a
rich space of commonsense inferences anchored to knowledge about specific
entities: for example, deciding the truthfulness of a claim "Harry Potter can
teach classes on how to fly on a broomstick." Can models learn to combine
entity knowledge with commonsense reasoning in this fashion? We introduce
CREAK, a testbed for commonsense reasoning about entity knowledge, bridging
fact-checking about entities (Harry Potter is a wizard and is skilled at riding
a broomstick) with commonsense inferences (if you're good at a skill you can
teach others how to do it). Our dataset consists of 13k human-authored English
claims about entities that are either true or false, in addition to a small
contrast set. Crowdworkers can easily come up with these statements and human
performance on the dataset is high (high 90s); we argue that models should be
able to blend entity knowledge and commonsense reasoning to do well here. In
our experiments, we focus on the closed-book setting and observe that a
baseline model finetuned on existing fact verification benchmark struggles on
CREAK. Training a model on CREAK improves accuracy by a substantial margin, but
still falls short of human performance. Our benchmark provides a unique probe
into natural language understanding models, testing both its ability to
retrieve facts (e.g., who teaches at the University of Chicago?) and unstated
commonsense knowledge (e.g., butlers do not yell at guests).
