  Transformer hugely benefits from its key design of the multi-head
self-attention network (SAN), which extracts information from various
perspectives through transforming the given input into different subspaces.
However, its simple linear transformation aggregation strategy may still
potentially fail to fully capture deeper contextualized information. In this
paper, we thus propose the capsule-Transformer, which extends the linear
transformation into a more general capsule routing algorithm by taking SAN as a
special case of capsule network. So that the resulted capsule-Transformer is
capable of obtaining a better attention distribution representation of the
input sequence via information aggregation among different heads and words.
Specifically, we see groups of attention weights in SAN as low layer capsules.
By applying the iterative capsule routing algorithm they can be further
aggregated into high layer capsules which contain deeper contextualized
information. Experimental results on the widely-used machine translation
datasets show our proposed capsule-Transformer outperforms strong Transformer
baseline significantly.
