  The effect of translationese has been studied in the field of machine
translation (MT), mostly with respect to training data. We study in depth the
effect of translationese on test data, using the test sets from the last three
editions of WMT's news shared task, containing 17 translation directions. We
show evidence that (i) the use of translationese in test sets results in
inflated human evaluation scores for MT systems; (ii) in some cases system
rankings do change and (iii) the impact translationese has on a translation
direction is inversely correlated to the translation quality attainable by
state-of-the-art MT systems for that direction.
