  Linear position interpolation helps pre-trained models using rotary position
embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using
linear position interpolation to extend the extrapolation range of models using
Attention with Linear Biases (ALiBi). We find position interpolation
significantly improves extrapolation capability on upstream language modelling
and downstream summarization and retrieval tasks.
