  In Neural Machine Translation, it is typically assumed that the sentence with
the highest estimated probability should also be the translation with the
highest quality as measured by humans. In this work, we question this
assumption and show that model estimates and translation quality only vaguely
correlate. We apply Minimum Bayes Risk (MBR) decoding on unbiased samples to
optimize diverse automated metrics of translation quality as an alternative
inference strategy to beam search. Instead of targeting the hypotheses with the
highest model probability, MBR decoding extracts the hypotheses with the
highest estimated quality. Our experiments show that the combination of a
neural translation model with a neural reference-based metric, BLEURT, results
in significant improvement in human evaluations. This improvement is obtained
with translations different from classical beam-search output: these
translations have much lower model likelihood and are less favored by surface
metrics like BLEU.
