  Incorporating prior knowledge into pre-trained language models has proven to
be effective for knowledge-driven NLP tasks, such as entity typing and relation
extraction. Current pre-training procedures usually inject external knowledge
into models by using knowledge masking, knowledge fusion and knowledge
replacement. However, factual information contained in the input sentences have
not been fully mined, and the external knowledge for injecting have not been
strictly checked. As a result, the context information cannot be fully
exploited and extra noise will be introduced or the amount of knowledge
injected is limited. To address these issues, we propose MLRIP, which modifies
the knowledge masking strategies proposed by ERNIE-Baidu, and introduce a
two-stage entity replacement strategy. Extensive experiments with comprehensive
analyses illustrate the superiority of MLRIP over BERT-based models in military
knowledge-driven NLP tasks.
