  Machine translation is a popular test bed for research in neural
sequence-to-sequence models but despite much recent research, there is still a
lack of understanding of these models. Practitioners report performance
degradation with large beams, the under-estimation of rare words and a lack of
diversity in the final translations. Our study relates some of these issues to
the inherent uncertainty of the task, due to the existence of multiple valid
translations for a single source sentence, and to the extrinsic uncertainty
caused by noisy training data. We propose tools and metrics to assess how
uncertainty in the data is captured by the model distribution and how it
affects search strategies that generate translations. Our results show that
search works remarkably well but that models tend to spread too much
probability mass over the hypothesis space. Next, we propose tools to assess
model calibration and show how to easily fix some shortcomings of current
models. As part of this study, we release multiple human reference translations
for two popular benchmarks.
