  For evaluating generation systems, automatic metrics such as BLEU cost
nothing to run but have been shown to correlate poorly with human judgment,
leading to systematic bias against certain model improvements. On the other
hand, averaging human judgments, the unbiased gold standard, is often too
expensive. In this paper, we use control variates to combine automatic metrics
with human evaluation to obtain an unbiased estimator with lower cost than
human evaluation alone. In practice, however, we obtain only a 7-13% cost
reduction on evaluating summarization and open-response question answering
systems. We then prove that our estimator is optimal: there is no unbiased
estimator with lower cost. Our theory further highlights the two fundamental
bottlenecks---the automatic metric and the prompt shown to human
evaluators---both of which need to be improved to obtain greater cost savings.
