  Human ratings are the gold standard in NLG evaluation. The standard protocol
is to collect ratings of generated text, average across annotators, and rank
NLG systems by their average scores. However, little consideration has been
given as to whether this approach faithfully captures human preferences.
Analyzing this standard protocol through the lens of utility theory in
economics, we identify the implicit assumptions it makes about annotators.
These assumptions are often violated in practice, in which case annotator
ratings cease to reflect their preferences. The most egregious violations come
from using Likert scales, which provably reverse the direction of the true
preference in certain cases. We suggest improvements to the standard protocol
to make it more theoretically sound, but even in its improved form, it cannot
be used to evaluate open-ended tasks like story generation. For the latter, we
propose a new human evaluation protocol called $\textit{system-level
probabilistic assessment}$ (SPA). When human evaluation of stories is done with
SPA, we can recover the ordering of GPT-3 models by size, with statistically
significant results. However, when human evaluation is done with the standard
protocol, less than half of the expected preferences can be recovered (e.g.,
there is no significant difference between $\texttt{curie}$ and
$\texttt{davinci}$, despite using a highly powered test).
