  It has been over six years since the Transformer architecture was put
forward. Surprisingly, the vanilla Transformer architecture is still widely
used today. One reason is that the lack of deep understanding and comprehensive
interpretation of the Transformer architecture makes it more challenging to
improve the Transformer architecture. In this paper, we first interpret the
Transformer architecture comprehensively in plain words based on our
understanding and experiences. The interpretations are further proved and
verified. These interpretations also cover the Extractor, a family of drop-in
replacements for the multi-head self-attention in the Transformer architecture.
Then, we propose an improvement on a type of the Extractor that outperforms the
self-attention, without introducing additional trainable parameters.
Experimental results demonstrate that the improved Extractor performs even
better, showing a way to improve the Transformer architecture.
