  By focusing the pre-training process on domain-specific corpora, some
domain-specific pre-trained language models (PLMs) have achieved
state-of-the-art results. However, it is under-investigated to design a unified
paradigm to inject domain knowledge in the PLM fine-tuning stage. We propose
KnowledgeDA, a unified domain language model development service to enhance the
task-specific training procedure with domain knowledge graphs. Given
domain-specific task texts input, KnowledgeDA can automatically generate a
domain-specific language model following three steps: (i) localize domain
knowledge entities in texts via an embedding-similarity approach; (ii) generate
augmented samples by retrieving replaceable domain entity pairs from two views
of both knowledge graph and training data; (iii) select high-quality augmented
samples for fine-tuning via confidence-based assessment. We implement a
prototype of KnowledgeDA to learn language models for two domains, healthcare
and software development. Experiments on domain-specific text classification
and QA tasks verify the effectiveness and generalizability of KnowledgeDA.
