  At the heart of improving conversational AI is the open problem of how to
evaluate conversations. Issues with automatic metrics are well known (Liu et
al., 2016, arXiv:1603.08023), with human evaluations still considered the gold
standard. Unfortunately, how to perform human evaluations is also an open
problem: differing data collection methods have varying levels of human
agreement and statistical sensitivity, resulting in differing amounts of human
annotation hours and labor costs. In this work we compare five different
crowdworker-based human evaluation methods and find that different methods are
best depending on the types of models compared, with no clear winner across the
board. While this highlights the open problems in the area, our analysis leads
to advice of when to use which one, and possible future directions.
