  External knowledge,e.g., entities and entity descriptions, can help humans
understand texts. Many works have been explored to include external knowledge
in the pre-trained models. These methods, generally, design pre-training tasks
and implicitly introduce knowledge by updating model weights, alternatively,
use it straightforwardly together with the original text. Though effective,
there are some limitations. On the one hand, it is implicit and only model
weights are paid attention to, the pre-trained entity embeddings are ignored.
On the other hand, entity descriptions may be lengthy, and inputting into the
model together with the original text may distract the model's attention. This
paper aims to explicitly include both entities and entity descriptions in the
fine-tuning stage. First, the pre-trained entity embeddings are fused with the
original text representation and updated by the backbone model layer by layer.
Second, descriptions are represented by the knowledge module outside the
backbone model, and each knowledge layer is selectively connected to one
backbone layer for fusing. Third, two knowledge-related auxiliary tasks, i.e.,
entity/description enhancement and entity enhancement/pollution task, are
designed to smooth the semantic gaps among evolved representations. We
conducted experiments on four knowledge-oriented tasks and two common tasks,
and the results achieved new state-of-the-art on several datasets. Besides, we
conduct an ablation study to show that each module in our method is necessary.
The code is available at https://github.com/lshowway/Ered.
