  Motivated by the success of pre-trained language models such as BERT in a
broad range of natural language processing (NLP) tasks, recent research efforts
have been made for adapting these models for different application domains.
Along this line, existing domain-oriented models have primarily followed the
vanilla BERT architecture and have a straightforward use of the domain corpus.
However, domain-oriented tasks usually require accurate understanding of domain
phrases, and such fine-grained phrase-level knowledge is hard to be captured by
existing pre-training scheme. Also, the word co-occurrences guided semantic
learning of pre-training models can be largely augmented by entity-level
association knowledge. But meanwhile, by doing so there is a risk of
introducing noise due to the lack of groundtruth word-level alignment. To
address the above issues, we provide a generalized domain-oriented approach,
which leverages auxiliary domain knowledge to improve the existing pre-training
framework from two aspects. First, to preserve phrase knowledge effectively, we
build a domain phrase pool as auxiliary training tool, meanwhile we introduce
Adaptive Hybrid Masked Model to incorporate such knowledge. It integrates two
learning modes, word learning and phrase learning, and allows them to switch
between each other. Second, we introduce Cross Entity Alignment to leverage
entity association as weak supervision to augment the semantic learning of
pre-trained models. To alleviate the potential noise in this process, we
introduce an interpretable Optimal Transport based approach to guide alignment
learning. Experiments on four domain-oriented tasks demonstrate the superiority
of our framework.
