  Length extrapolation has attracted considerable attention recently since it
allows transformers to be tested on longer sequences than those used in
training. Previous research has shown that this property can be attained by
using carefully designed Relative Positional Encodings (RPEs). While these
methods perform well on a variety of corpora, the conditions for length
extrapolation have yet to be investigated. This paper attempts to determine
what types of RPEs allow for length extrapolation through a thorough
mathematical and empirical analysis. We discover that a transformer is certain
to possess this property as long as the series that corresponds to the RPE's
exponential converges. Two practices are derived from the conditions and
examined in language modeling tasks on a variety of corpora. As a bonus from
the conditions, we derive a new Theoretical Receptive Field (TRF) to measure
the receptive field of RPEs without taking any training steps. Extensive
experiments are conducted on the Wikitext-103, Books, Github, and WikiBook
datasets to demonstrate the viability of our discovered conditions. We also
compare TRF to Empirical Receptive Field (ERF) across different models, showing
consistently matched trends on the aforementioned datasets. The code is
available at https://github.com/OpenNLPLab/Rpe.
