  The field of machine translation faces an under-recognized problem because of
inconsistency in the reporting of scores from its dominant metric. Although
people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose
values can vary wildly with changes to these parameters. These parameters are
often not reported or are hard to find, and consequently, BLEU scores between
papers cannot be directly compared. I quantify this variation, finding
differences as high as 1.8 between commonly used configurations. The main
culprit is different tokenization and normalization schemes applied to the
reference. Pointing to the success of the parsing community, I suggest machine
translation researchers settle upon the BLEU scheme used by the annual
Conference on Machine Translation (WMT), which does not allow for user-supplied
reference processing, and provide a new tool, SacreBLEU, to facilitate this.
