  Precisely assessing the progress in natural language generation (NLG) tasks
is challenging, and human evaluation to establish a preference in a model's
output over another is often necessary. However, human evaluation is usually
costly, difficult to reproduce, and non-reusable. In this paper, we propose a
new and simple automatic evaluation method for NLG called Near-Negative
Distinction (NND) that repurposes prior human annotations into NND tests. In an
NND test, an NLG model must place a higher likelihood on a high-quality output
candidate than on a near-negative candidate with a known error. Model
performance is established by the number of NND tests a model passes, as well
as the distribution over task-specific errors the model fails on. Through
experiments on three NLG tasks (question generation, question answering, and
summarization), we show that NND achieves a higher correlation with human
judgments than standard NLG evaluation metrics. We then illustrate NND
evaluation in four practical scenarios, for example performing fine-grain model
analysis, or studying model training dynamics. Our findings suggest that NND
can give a second life to human annotations and provide low-cost NLG
evaluation.
