  Currently, contextualized word representations are learned by intricate
neural network models, such as masked neural language models (MNLMs). The new
representations significantly enhanced the performance in automated question
answering by reading paragraphs. However, identifying the detailed knowledge
trained in the MNLMs is difficult owing to numerous and intermingled
parameters. This paper provides empirical but insightful analyses on the
pretrained MNLMs with respect to common sense knowledge. First, we propose a
test that measures what types of common sense knowledge do pretrained MNLMs
understand. From the test, we observed that MNLMs partially understand various
types of common sense knowledge but do not accurately understand the semantic
meaning of relations. In addition, based on the difficulty of the
question-answering task problems, we observed that pretrained MLM-based models
are still vulnerable to problems that require common sense knowledge. We also
experimentally demonstrated that we can elevate existing MNLM-based models by
combining knowledge from an external common sense repository.
