  Meaning is defined by the company it keeps. However, company is two-fold:
It's based on the identity of tokens and also on their position (topology). We
argue that a position-centric perspective is more general and useful. The
classic MLM and CLM objectives in NLP are easily phrased as position
predictions over the whole vocabulary. Adapting the relative position encoding
paradigm in NLP to create relative labels for self-supervised learning, we seek
to show superior pre-training judged by performance on downstream tasks.
