  Multi-head attention advances neural machine translation by working out
multiple versions of attention in different subspaces, but the neglect of
semantic overlapping between subspaces increases the difficulty of translation
and consequently hinders the further improvement of translation performance. In
this paper, we employ capsule networks to comb the information from the
multiple heads of the attention so that similar information can be clustered
and unique information can be reserved. To this end, we adopt two routing
mechanisms of Dynamic Routing and EM Routing, to fulfill the clustering and
separating. We conducted experiments on Chinese-to-English and
English-to-German translation tasks and got consistent improvements over the
strong Transformer baseline.
