  The performance of neural machine translation systems is commonly evaluated
in terms of BLEU. However, due to its reliance on target language properties
and generation, the BLEU metric does not allow an assessment of which
translation directions are more difficult to model. In this paper, we propose
cross-mutual information (XMI): an asymmetric information-theoretic metric of
machine translation difficulty that exploits the probabilistic nature of most
neural machine translation models. XMI allows us to better evaluate the
difficulty of translating text into the target language while controlling for
the difficulty of the target-side generation component independent of the
translation task. We then present the first systematic and controlled study of
cross-lingual translation difficulties using modern neural translation systems.
Code for replicating our experiments is available online at
https://github.com/e-bug/nmt-difficulty.
