  Progress on commonsense reasoning is usually measured from performance
improvements on Question Answering tasks designed to require commonsense
knowledge. However, fine-tuning large Language Models (LMs) on these specific
tasks does not directly evaluate commonsense learned during pre-training. The
most direct assessments of commonsense knowledge in pre-trained LMs are
arguably cloze-style tasks targeting commonsense assertions (e.g., A pen is
used for [MASK].). However, this approach is restricted by the LM's vocabulary
available for masked predictions, and its precision is subject to the context
provided by the assertion. In this work, we present a method for enriching LMs
with a grounded sense inventory (i.e., WordNet) available at the vocabulary
level, without further training. This modification augments the prediction
space of cloze-style prompts to the size of a large ontology while enabling
finer-grained (sense-level) queries and predictions. In order to evaluate LMs
with higher precision, we propose SenseLAMA, a cloze-style task featuring
verbalized relations from disambiguated triples sourced from WordNet, WikiData,
and ConceptNet. Applying our method to BERT, producing a WordNet-enriched
version named SynBERT, we find that LMs can learn non-trivial commonsense
knowledge from self-supervision, covering numerous relations, and more
effectively than comparable similarity-based approaches.
