  In MT evaluation, pairwise comparisons are conducted to identify the better
system. In conducting the comparison, the experimenter must allocate a budget
to collect Direct Assessment (DA) judgments. We provide a cost effective way to
spend the budget, but show that typical budget sizes often do not allow for
solid comparison. Taking the perspective that the basis of solid comparison is
in achieving statistical significance, we study the power (rate of achieving
significance) on a large collection of pairwise DA comparisons. Due to the
nature of statistical estimation, power is low for differentiating less than
1-2 DA points, and to achieve a notable increase in power requires at least
2-3x more samples. Applying variance reduction alone will not yield these
gains, so we must face the reality of undetectable differences and spending
increases. In this context, we propose interim testing, an "early stopping"
collection procedure that yields more power per judgment collected, which
adaptively focuses the budget on pairs that are borderline significant. Interim
testing can achieve up to a 27% efficiency gain when spending 3x the current
budget, or 18% savings at the current evaluation power.
