  Pre-trained language models (PLMs) cannot well recall rich factual knowledge
of entities exhibited in large-scale corpora, especially those rare entities.
In this paper, we propose to build a simple but effective Pluggable Entity
Lookup Table (PELT) on demand by aggregating the entity's output
representations of multiple occurrences in the corpora. PELT can be compatibly
plugged as inputs to infuse supplemental entity knowledge into PLMs. Compared
to previous knowledge-enhanced PLMs, PELT only requires 0.2%-5% pre-computation
with capability of acquiring knowledge from out-of-domain corpora for domain
adaptation scenario. The experiments on knowledge-related tasks demonstrate
that our method, PELT, can flexibly and effectively transfer entity knowledge
from related corpora into PLMs with different architectures.
