  The Transformer translation model is based on the multi-head attention
mechanism, which can be parallelized easily. The multi-head attention network
performs the scaled dot-product attention function in parallel, empowering the
model by jointly attending to information from different representation
subspaces at different positions. In this paper, we present an approach to
learning a hard retrieval attention where an attention head only attends to one
token in the sentence rather than all tokens. The matrix multiplication between
attention probabilities and the value sequence in the standard scaled
dot-product attention can thus be replaced by a simple and efficient retrieval
operation. We show that our hard retrieval attention mechanism is 1.43 times
faster in decoding, while preserving translation quality on a wide range of
machine translation tasks when used in the decoder self- and cross-attention
networks.
