  This paper considers the problem of zero-shot entity linking, in which a link
in the test time may not present in training. Following the prevailing
BERT-based research efforts, we find a simple yet effective way is to expand
the long-range sequence modeling. Unlike many previous methods, our method does
not require expensive pre-training of BERT with long position embedding.
Instead, we propose an efficient position embeddings initialization method
called Embedding-repeat, which initializes larger position embeddings based on
BERT-Base. On Wikia's zero-shot EL dataset, our method improves the SOTA from
76.06% to 79.08%, and for its long data, the corresponding improvement is from
74.57% to 82.14%. Our experiments suggest the effectiveness of long-range
sequence modeling without retraining the BERT model.
