  Transformer has taken the field of natural language processing (NLP) by storm
since its birth. Further, Large language models (LLMs) built upon it have
captured worldwide attention due to its superior abilities. Nevertheless, all
Transformer-based models including these powerful LLMs suffer from a preset
length limit and can hardly generalize from short training sequences to longer
inference ones, namely, they can not perform length extrapolation. Hence, a
plethora of methods have been proposed to enhance length extrapolation of
Transformer, in which the positional encoding (PE) is recognized as the major
factor. In this survey, we present these advances towards length extrapolation
in a unified notation from the perspective of PE. Specifically, we first
introduce extrapolatable PEs, including absolute and relative PEs. Then, we
dive into extrapolation methods based on them, covering position interpolation
and randomized position methods. Finally, several challenges and future
directions in this area are highlighted. Through this survey, We aim to enable
the reader to gain a deep understanding of existing methods and provide stimuli
for future research.
