  Confidence calibration, which aims to make model predictions equal to the
true correctness measures, is important for neural machine translation (NMT)
because it is able to offer useful indicators of translation errors in the
generated output. While prior studies have shown that NMT models trained with
label smoothing are well-calibrated on the ground-truth training data, we find
that miscalibration still remains a severe challenge for NMT during inference
due to the discrepancy between training and inference. By carefully designing
experiments on three language pairs, our work provides in-depth analyses of the
correlation between calibration and translation performance as well as
linguistic properties of miscalibration and reports a number of interesting
findings that might help humans better analyze, understand and improve NMT
models. Based on these observations, we further propose a new graduated label
smoothing method that can improve both inference calibration and translation
performance.
