  While often assumed a gold standard, effective human evaluation of text
generation remains an important, open area for research. We revisit this
problem with a focus on producing consistent evaluations that are reproducible
-- over time and across different populations. We study this goal in different
stages of the human evaluation pipeline. In particular, we consider design
choices for the annotation interface used to elicit human judgments and their
impact on reproducibility. Furthermore, we develop an automated mechanism for
maintaining annotator quality via a probabilistic model that detects and
excludes noisy annotators. Putting these lessons together, we introduce GENIE:
a system for running standardized human evaluations across different generation
tasks. We instantiate GENIE with datasets representing four core challenges in
text generation: machine translation, summarization, commonsense reasoning, and
machine comprehension. For each task, GENIE offers a leaderboard that
automatically crowdsources annotations for submissions, evaluating them along
axes such as correctness, conciseness, and fluency. We have made the GENIE
leaderboards publicly available, and have already ranked 50 submissions from 10
different research groups. We hope GENIE encourages further progress toward
effective, standardized evaluations for text generation.
