  Recently, Transformer has achieved the state-of-the-art performance on many
machine translation tasks. However, without syntax knowledge explicitly
considered in the encoder, incorrect context information that violates the
syntax structure may be integrated into source hidden states, leading to
erroneous translations. In this paper, we propose a novel method to incorporate
source dependencies into the Transformer. Specifically, we adopt the source
dependency tree and define two matrices to represent the dependency relations.
Based on the matrices, two heads in the multi-head self-attention module are
trained in a supervised manner and two extra cross entropy losses are
introduced into the training objective function. Under this training objective,
the model is trained to learn the source dependency relations directly. Without
requiring pre-parsed input during inference, our model can generate better
translations with the dependency-aware context information. Experiments on
bi-directional Chinese-to-English, English-to-Japanese and English-to-German
translation tasks show that our proposed method can significantly improve the
Transformer baseline.
