  Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches
their representations and justifies their use as knowledge bases. Previous work
has focused on probing PLMs for factual knowledge by measuring how often they
can correctly predict an object entity given a subject and a relation, and
improving fact retrieval by optimizing the prompts used for querying PLMs. In
this work, we consider a complementary aspect, namely the coherency of factual
knowledge in PLMs, i.e., how often can PLMs predict the subject entity given
its initial prediction of the object entity. This goes beyond evaluating how
much PLMs know, and focuses on the internal state of knowledge inside them. Our
results indicate that PLMs have low coherency using manually written, optimized
and paraphrased prompts, but including an evidence paragraph leads to
substantial improvement. This shows that PLMs fail to model inverse relations
and need further enhancements to be able to handle retrieving facts from their
parameters in a coherent manner, and to be considered as knowledge bases.
