  Pretrained language models can be queried for factual knowledge, with
potential applications in knowledge base acquisition and tasks that require
inference. However, for that, we need to know how reliable this knowledge is,
and recent work has shown that monolingual English language models lack
consistency when predicting factual knowledge, that is, they fill-in-the-blank
differently for paraphrases describing the same fact. In this paper, we extend
the analysis of consistency to a multilingual setting. We introduce a resource,
mParaRel, and investigate (i) whether multilingual language models such as
mBERT and XLM-R are more consistent than their monolingual counterparts; and
(ii) if such models are equally consistent across languages. We find that mBERT
is as inconsistent as English BERT in English paraphrases, but that both mBERT
and XLM-R exhibit a high degree of inconsistency in English and even more so
for all the other 45 languages.
