  Several uncertainty estimation methods have been recently proposed for
machine translation evaluation. While these methods can provide a useful
indication of when not to trust model predictions, we show in this paper that
the majority of them tend to underestimate model uncertainty, and as a result
they often produce misleading confidence intervals that do not cover the ground
truth. We propose as an alternative the use of conformal prediction, a
distribution-free method to obtain confidence intervals with a theoretically
established guarantee on coverage. First, we demonstrate that split conformal
prediction can ``correct'' the confidence intervals of previous methods to
yield a desired coverage level. Then, we highlight biases in estimated
confidence intervals, both in terms of the translation language pairs and the
quality of translations. We apply conditional conformal prediction techniques
to obtain calibration subsets for each data subgroup, leading to equalized
coverage.
