  Language models (LMs) show state of the art performance for common sense (CS)
question answering, but whether this ability implies a human-level mastery of
CS remains an open question. Understanding the limitations and strengths of LMs
can help researchers improve these models, potentially by developing novel ways
of integrating external CS knowledge. We devise a series of tests and
measurements to systematically quantify their performance on different aspects
of CS. We propose the use of cloze testing combined with word embeddings to
measure the LM's robustness and confidence. Our results show than although
language models tend to achieve human-like accuracy, their confidence is
subpar. Future work can leverage this information to build more complex
systems, such as an ensemble of symbolic and distributed knowledge.
