  As we show in this paper, the prediction for output token $n+1$ of
Transformer architectures without one of the mechanisms of positional encodings
and causal attention is invariant to permutations of input tokens $1, 2, ...,
n-1$. Usually, both mechanisms are employed and the symmetry with respect to
the input tokens is broken. Recently, it has been shown that one can train
Transformers without positional encodings. This must be enabled by the causal
attention mechanism. In this paper, we elaborate on the argument that the
causal connection mechanism must be responsible for the fact that Transformers
are able to model input sequences where the order is important. Vertical
"slices" of Transformers are all encouraged to represent the same location $k$
in the input sequence. We hypothesize that residual connections contribute to
this phenomenon, and demonstrate evidence for this.
