  Knowledge Bases (KBs) are easy to query, verifiable, and interpretable. They
however scale with man-hours and high-quality data. Masked Language Models
(MLMs), such as BERT, scale with computing power as well as unstructured raw
text data. The knowledge contained within those models is however not directly
interpretable. We propose to perform link prediction with MLMs to address both
the KBs scalability issues and the MLMs interpretability issues. To do that we
introduce MLMLM, Mean Likelihood Masked Language Model, an approach comparing
the mean likelihood of generating the different entities to perform link
prediction in a tractable manner. We obtain State of the Art (SotA) results on
the WN18RR dataset and the best non-entity-embedding based results on the
FB15k-237 dataset. We also obtain convincing results on link prediction on
previously unseen entities, making MLMLM a suitable approach to introducing new
entities to a KB.
