  Though machine translation errors caused by the lack of context beyond one
sentence have long been acknowledged, the development of context-aware NMT
systems is hampered by several problems. Firstly, standard metrics are not
sensitive to improvements in consistency in document-level translations.
Secondly, previous work on context-aware NMT assumed that the sentence-aligned
parallel data consisted of complete documents while in most practical scenarios
such document-level data constitutes only a fraction of the available parallel
data. To address the first issue, we perform a human study on an
English-Russian subtitles dataset and identify deixis, ellipsis and lexical
cohesion as three main sources of inconsistency. We then create test sets
targeting these phenomena. To address the second shortcoming, we consider a
set-up in which a much larger amount of sentence-level data is available
compared to that aligned at the document level. We introduce a model that is
suitable for this scenario and demonstrate major gains over a context-agnostic
baseline on our new benchmarks without sacrificing performance as measured with
BLEU.
