  Many contextualized word representations are now learned by intricate neural
network models, such as masked neural language models (MNLMs) which are made up
of huge neural network structures and trained to restore the masked text. Such
representations demonstrate superhuman performance in some reading
comprehension (RC) tasks which extract a proper answer in the context given a
question. However, identifying the detailed knowledge trained in MNLMs is
challenging owing to numerous and intermingled model parameters. This paper
provides new insights and empirical analyses on commonsense knowledge included
in pretrained MNLMs. First, we use a diagnostic test that evaluates whether
commonsense knowledge is properly trained in MNLMs. We observe that a large
proportion of commonsense knowledge is not appropriately trained in MNLMs and
MNLMs do not often understand the semantic meaning of relations accurately. In
addition, we find that the MNLM-based RC models are still vulnerable to
semantic variations that require commonsense knowledge. Finally, we discover
the fundamental reason why some knowledge is not trained. We further suggest
that utilizing an external commonsense knowledge repository can be an effective
solution. We exemplify the possibility to overcome the limitations of the
MNLM-based RC models by enriching text with the required knowledge from an
external commonsense knowledge repository in controlled experiments.
