  It is often observed in knowledge-centric tasks (e.g., common sense question
and answering, relation classification) that the integration of external
knowledge such as entity representation into language models can help provide
useful information to boost the performance. However, it is still unclear
whether this benefit can extend to general natural language understanding (NLU)
tasks. In this work, we empirically investigated the contribution of external
knowledge by measuring the end-to-end performance of language models with
various knowledge integration methods. We find that the introduction of
knowledge can significantly improve the results on certain tasks while having
no adverse effects on other tasks. We then employ mutual information to reflect
the difference brought by knowledge and a neural interpretation model to reveal
how a language model utilizes external knowledge. Our study provides valuable
insights and guidance for practitioners to equip NLP models with knowledge.
