  We aim to shed light on the strengths and weaknesses of the newly introduced
neural machine translation paradigm. To that end, we conduct a multifaceted
evaluation in which we compare outputs produced by state-of-the-art neural
machine translation and phrase-based machine translation systems for 9 language
directions across a number of dimensions. Specifically, we measure the
similarity of the outputs, their fluency and amount of reordering, the effect
of sentence length and performance across different error categories. We find
out that translations produced by neural machine translation systems are
considerably different, more fluent and more accurate in terms of word order
compared to those produced by phrase-based systems. Neural machine translation
systems are also more accurate at producing inflected forms, but they perform
poorly when translating very long sentences.
