  Recently, the performance of Pre-trained Language Models (PLMs) has been
significantly improved by injecting knowledge facts to enhance their abilities
of language understanding. For medical domains, the background knowledge
sources are especially useful, due to the massive medical terms and their
complicated relations are difficult to understand in text. In this work, we
introduce SMedBERT, a medical PLM trained on large-scale medical corpora,
incorporating deep structured semantic knowledge from neighbors of
linked-entity.In SMedBERT, the mention-neighbor hybrid attention is proposed to
learn heterogeneous-entity information, which infuses the semantic
representations of entity types into the homogeneous neighboring entity
structure. Apart from knowledge integration as external features, we propose to
employ the neighbors of linked-entities in the knowledge graph as additional
global contexts of text mentions, allowing them to communicate via shared
neighbors, thus enrich their semantic representations. Experiments demonstrate
that SMedBERT significantly outperforms strong baselines in various
knowledge-intensive Chinese medical tasks. It also improves the performance of
other tasks such as question answering, question matching and natural language
inference.
