  Kendall's tau is frequently used to meta-evaluate how well machine
translation (MT) evaluation metrics score individual translations. Its focus on
pairwise score comparisons is intuitive but raises the question of how ties
should be handled, a gray area that has motivated different variants in the
literature. We demonstrate that, in settings like modern MT meta-evaluation,
existing variants have weaknesses arising from their handling of ties, and in
some situations can even be gamed. We propose instead to meta-evaluate metrics
with a version of pairwise accuracy that gives metrics credit for correctly
predicting ties, in combination with a tie calibration procedure that
automatically introduces ties into metric scores, enabling fair comparison
between metrics that do and do not predict ties. We argue and provide
experimental evidence that these modifications lead to fairer ranking-based
assessments of metric performance.
