  Natural language generation (NLG) systems are commonly evaluated using n-gram
overlap measures (e.g. BLEU, ROUGE). These measures do not directly capture
semantics or speaker intentions, and so they often turn out to be misaligned
with our true goals for NLG. In this work, we argue instead for
communication-based evaluations: assuming the purpose of an NLG system is to
convey information to a reader/listener, we can directly evaluate its
effectiveness at this task using the Rational Speech Acts model of pragmatic
language use. We illustrate with a color reference dataset that contains
descriptions in pre-defined quality categories, showing that our method better
aligns with these quality categories than do any of the prominent n-gram
overlap methods.
