  Causal transformer language models (LMs), such as GPT-3, typically require
some form of positional encoding, such as positional embeddings. However, we
show that LMs without any explicit positional encoding are still competitive
with standard models, and that this phenomenon is robust across different
datasets, model sizes, and sequence lengths. Probing experiments reveal that
such models acquire an implicit notion of absolute positions throughout the
network, effectively compensating for the missing information. We conjecture
that causal attention enables the model to infer the number of predecessors
that each token can attend to, thereby approximating its absolute position. Our
findings indicate that causal LMs might derive positional awareness not only
from the explicit positioning mechanism, but also from the effects of the
causal mask.
