  Pretrained Language Models (PLM) have established a new paradigm through
learning informative contextualized representations on large-scale text corpus.
This new paradigm has revolutionized the entire field of natural language
processing, and set the new state-of-the-art performance for a wide variety of
NLP tasks. However, though PLMs could store certain knowledge/facts from
training corpus, their knowledge awareness is still far from satisfactory. To
address this issue, integrating knowledge into PLMs have recently become a very
active research area and a variety of approaches have been developed. In this
paper, we provide a comprehensive survey of the literature on this emerging and
fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs).
We introduce three taxonomies to categorize existing work. Besides, we also
survey the various NLU and NLG applications on which KE-PLM has demonstrated
superior performance over vanilla PLMs. Finally, we discuss challenges that
face KE-PLMs and also promising directions for future research.
