  Length generalization, the ability to generalize from small training context
sizes to larger ones, is a critical challenge in the development of
Transformer-based language models. Positional encoding (PE) has been identified
as a major factor influencing length generalization, but the exact impact of
different PE schemes on extrapolation in downstream tasks remains unclear. In
this paper, we conduct a systematic empirical study comparing the length
generalization performance of decoder-only Transformers with five different
position encoding approaches including Absolute Position Embedding (APE), T5's
Relative PE, ALiBi, and Rotary, in addition to Transformers without positional
encoding (NoPE). Our evaluation encompasses a battery of reasoning and
mathematical tasks. Our findings reveal that the most commonly used positional
encoding methods, such as ALiBi, Rotary, and APE, are not well suited for
length generalization in downstream tasks. More importantly, NoPE outperforms
other explicit positional encoding methods while requiring no additional
computation. We theoretically demonstrate that NoPE can represent both absolute
and relative PEs, but when trained with SGD, it mostly resembles T5's relative
PE attention patterns. Finally, we find that scratchpad is not always helpful
to solve length generalization and its format highly impacts the model's
performance. Overall, our work suggests that explicit position embeddings are
not essential for decoder-only Transformers to generalize well to longer
sequences.
