  Several neural-based metrics have been recently proposed to evaluate machine
translation quality. However, all of them resort to point estimates, which
provide limited information at segment level. This is made worse as they are
trained on noisy, biased and scarce human judgements, often resulting in
unreliable quality predictions. In this paper, we introduce uncertainty-aware
MT evaluation and analyze the trustworthiness of the predicted quality. We
combine the COMET framework with two uncertainty estimation methods, Monte
Carlo dropout and deep ensembles, to obtain quality scores along with
confidence intervals. We compare the performance of our uncertainty-aware MT
evaluation methods across multiple language pairs from the QT21 dataset and the
WMT20 metrics task, augmented with MQM annotations. We experiment with varying
numbers of references and further discuss the usefulness of uncertainty-aware
quality estimation (without references) to flag possibly critical translation
mistakes.
