  In Transformer-based architectures, the attention mechanism is inherently
permutation-invariant with respect to the input sequence's tokens. To impose
sequential order, token positions are typically encoded using a scheme with
either fixed or learnable parameters. We introduce Hyperbolic Positional
Encoding (HyPE), a novel method that utilizes hyperbolic functions' properties
to encode tokens' relative positions. This approach biases the attention
mechanism without the necessity of storing the $O(L^2)$ values of the mask,
with $L$ being the length of the input sequence. HyPE leverages preliminary
concatenation operations and matrix multiplications, facilitating the encoding
of relative distances indirectly incorporating biases into the softmax
computation. This design ensures compatibility with FlashAttention-2 and
supports the gradient backpropagation for any potential learnable parameters
within the encoding. We analytically demonstrate that, by careful
hyperparameter selection, HyPE can approximate the attention bias of ALiBi,
thereby offering promising generalization capabilities for contexts extending
beyond the lengths encountered during pretraining. The experimental evaluation
of HyPE is proposed as a direction for future research.
