  Knowledge-enhanced language representation learning has shown promising
results across various knowledge-intensive NLP tasks. However, prior methods
are limited in efficient utilization of multilingual knowledge graph (KG) data
for language model (LM) pretraining. They often train LMs with KGs in indirect
ways, relying on extra entity/relation embeddings to facilitate knowledge
injection. In this work, we explore methods to make better use of the
multilingual annotation and language agnostic property of KG triples, and
present novel knowledge based multilingual language models (KMLMs) trained
directly on the knowledge triples. We first generate a large amount of
multilingual synthetic sentences using the Wikidata KG triples. Then based on
the intra- and inter-sentence structures of the generated data, we design
pretraining tasks to enable the LMs to not only memorize the factual knowledge
but also learn useful logical patterns. Our pretrained KMLMs demonstrate
significant performance improvements on a wide range of knowledge-intensive
cross-lingual tasks, including named entity recognition (NER), factual
knowledge retrieval, relation classification, and a newly designed logical
reasoning task.
