  Recent advances in Transformer models allow for unprecedented sequence
lengths, due to linear space and time complexity. In the meantime, relative
positional encoding (RPE) was proposed as beneficial for classical Transformers
and consists in exploiting lags instead of absolute positions for inference.
Still, RPE is not available for the recent linear-variants of the Transformer,
because it requires the explicit computation of the attention matrix, which is
precisely what is avoided by such methods. In this paper, we bridge this gap
and present Stochastic Positional Encoding as a way to generate PE that can be
used as a replacement to the classical additive (sinusoidal) PE and provably
behaves like RPE. The main theoretical contribution is to make a connection
between positional encoding and cross-covariance structures of correlated
Gaussian processes. We illustrate the performance of our approach on the
Long-Range Arena benchmark and on music generation.
