  The Transformer architecture consists of self-attention and feed-forward
networks (FFNs) which can be viewed as key-value memories according to previous
works. However, FFN and traditional memory utilize different activation
functions (i.e., ReLU and Softmax respectively), which makes them not
equivalent. In this paper, we first rebuild the connections between FFN and
key-value memory by conducting extensive studies on ReLU and Softmax, and find
they are equivalent when adding an additional layer normalization module on
Softmax. In addition, ReLU outperforms Softmax on both FFN and key-value memory
when the number of value slots is large. We analyze the reasons and then
explore this good property of ReLU on the self-attention network where the
original Softmax activation performs poorly on long input sequences. We then
propose a full ReLU architecture named ReLUFormer which performs better than
the baseline Transformer on long sequence tasks such as document translation.
This paper sheds light on the following points: 1) Softmax and ReLU use
different normalization methods over elements which lead to different variances
of results, and ReLU is good at dealing with a large number of key-value slots;
2) FFN and key-value memory are equivalent, and thus the Transformer can be
viewed as a memory network where FFNs and self-attention networks are both
key-value memories.
