  Pretrained deep contextual representations have advanced the state-of-the-art
on various commonsense NLP tasks, but we lack a concrete understanding of the
capability of these models. Thus, we investigate and challenge several aspects
of BERT's commonsense representation abilities. First, we probe BERT's ability
to classify various object attributes, demonstrating that BERT shows a strong
ability in encoding various commonsense features in its embedding space, but is
still deficient in many areas. Next, we show that, by augmenting BERT's
pretraining data with additional data related to the deficient attributes, we
are able to improve performance on a downstream commonsense reasoning task
while using a minimal amount of data. Finally, we develop a method of
fine-tuning knowledge graphs embeddings alongside BERT and show the continued
importance of explicit knowledge graphs.
