  Directly training a document-to-document (Doc2Doc) neural machine translation
(NMT) via Transformer from scratch, especially on small datasets usually fails
to converge. Our dedicated probing tasks show that 1) both the absolute
position and relative position information gets gradually weakened or even
vanished once it reaches the upper encoder layers, and 2) the vanishing of
absolute position information in encoder output causes the training failure of
Doc2Doc NMT. To alleviate this problem, we propose a position-aware Transformer
(P-Transformer) to enhance both the absolute and relative position information
in both self-attention and cross-attention. Specifically, we integrate absolute
positional information, i.e., position embeddings, into the query-key pairs
both in self-attention and cross-attention through a simple yet effective
addition operation. Moreover, we also integrate relative position encoding in
self-attention. The proposed P-Transformer utilizes sinusoidal position
encoding and does not require any task-specified position embedding, segment
embedding, or attention mechanism. Through the above methods, we build a
Doc2Doc NMT model with P-Transformer, which ingests the source document and
completely generates the target document in a sequence-to-sequence (seq2seq)
way. In addition, P-Transformer can be applied to seq2seq-based
document-to-sentence (Doc2Sent) and sentence-to-sentence (Sent2Sent)
translation. Extensive experimental results of Doc2Doc NMT show that
P-Transformer significantly outperforms strong baselines on widely-used 9
document-level datasets in 7 language pairs, covering small-, middle-, and
large-scales, and achieves a new state-of-the-art. Experimentation on discourse
phenomena shows that our Doc2Doc NMT models improve the translation quality in
both BLEU and discourse coherence. We make our code available on Github.
