  The high-quality translation results produced by machine translation (MT)
systems still pose a huge challenge for automatic evaluation. Current MT
evaluation pays the same attention to each sentence component, while the
questions of real-world examinations (e.g., university examinations) have
different difficulties and weightings. In this paper, we propose a novel
difficulty-aware MT evaluation metric, expanding the evaluation dimension by
taking translation difficulty into consideration. A translation that fails to
be predicted by most MT systems will be treated as a difficult one and assigned
a large weight in the final score function, and conversely. Experimental
results on the WMT19 English-German Metrics shared tasks show that our proposed
method outperforms commonly used MT metrics in terms of human correlation. In
particular, our proposed method performs well even when all the MT systems are
very competitive, which is when most existing metrics fail to distinguish
between them. The source code is freely available at
https://github.com/NLP2CT/Difficulty-Aware-MT-Evaluation.
