  In this paper, we show that structures similar to self-attention are natural
to learn many sequence-to-sequence problems from the perspective of symmetry.
Inspired by language processing applications, we study the orthogonal
equivariance of seq2seq functions with knowledge, which are functions taking
two inputs -- an input sequence and a ``knowledge'' -- and outputting another
sequence. The knowledge consists of a set of vectors in the same embedding
space as the input sequence, containing the information of the language used to
process the input sequence. We show that orthogonal equivariance in the
embedding space is natural for seq2seq functions with knowledge, and under such
equivariance the function must take the form close to the self-attention. This
shows that network structures similar to self-attention are the right
structures to represent the target function of many seq2seq problems. The
representation can be further refined if a ``finite information principle'' is
considered, or a permutation equivariance holds for the elements of the input
sequence.
