  Trainable evaluation metrics for machine translation (MT) exhibit strong
correlation with human judgements, but they are often hard to interpret and
might produce unreliable scores under noisy or out-of-domain data. Recent work
has attempted to mitigate this with simple uncertainty quantification
techniques (Monte Carlo dropout and deep ensembles), however these techniques
(as we show) are limited in several ways -- for example, they are unable to
distinguish between different kinds of uncertainty, and they are time and
memory consuming. In this paper, we propose more powerful and efficient
uncertainty predictors for MT evaluation, and we assess their ability to target
different sources of aleatoric and epistemic uncertainty. To this end, we
develop and compare training objectives for the COMET metric to enhance it with
an uncertainty prediction output, including heteroscedastic regression,
divergence minimization, and direct uncertainty prediction. Our experiments
show improved results on uncertainty prediction for the WMT metrics task
datasets, with a substantial reduction in computational costs. Moreover, they
demonstrate the ability of these predictors to address specific uncertainty
causes in MT evaluation, such as low quality references and out-of-domain data.
