  Although the Transformer translation model (Vaswani et al., 2017) has
achieved state-of-the-art performance in a variety of translation tasks, how to
use document-level context to deal with discourse phenomena problematic for
Transformer still remains a challenge. In this work, we extend the Transformer
model with a new context encoder to represent document-level context, which is
then incorporated into the original encoder and decoder. As large-scale
document-level parallel corpora are usually not available, we introduce a
two-step training method to take full advantage of abundant sentence-level
parallel corpora and limited document-level parallel corpora. Experiments on
the NIST Chinese-English datasets and the IWSLT French-English datasets show
that our approach improves over Transformer significantly.
