  Recently, neural language representation models pre-trained on large corpus
can capture rich co-occurrence information and be fine-tuned in downstream
tasks to improve the performance. As a result, they have achieved
state-of-the-art results in a large range of language tasks. However, there
exists other valuable semantic information such as similar, opposite, or other
possible meanings in external knowledge graphs (KGs). We argue that entities in
KGs could be used to enhance the correct semantic meaning of language
sentences. In this paper, we propose a new method CKG: Dynamic Representation
Based on \textbf{C}ontext and \textbf{K}nowledge \textbf{G}raph. On the one
side, CKG can extract rich semantic information of large corpus. On the other
side, it can make full use of inside information such as co-occurrence in large
corpus and outside information such as similar entities in KGs. We conduct
extensive experiments on a wide range of tasks, including QQP, MRPC, SST-5,
SQuAD, CoNLL 2003, and SNLI. The experiment results show that CKG achieves SOTA
89.2 on SQuAD compared with SAN (84.4), ELMo (85.8), and BERT$_{Base}$ (88.5).
