  Evaluating open-domain dialogue systems is difficult due to the diversity of
possible correct answers. Automatic metrics such as BLEU correlate weakly with
human annotations, resulting in a significant bias across different models and
datasets. Some researchers resort to human judgment experimentation for
assessing response quality, which is expensive, time consuming, and not
scalable. Moreover, judges tend to evaluate a small number of dialogues,
meaning that minor differences in evaluation configuration may lead to
dissimilar results. In this paper, we present interpretable metrics for
evaluating topic coherence by making use of distributed sentence
representations. Furthermore, we introduce calculable approximations of human
judgment based on conversational coherence by adopting state-of-the-art
entailment techniques. Results show that our metrics can be used as a surrogate
for human judgment, making it easy to evaluate dialogue systems on large-scale
datasets and allowing an unbiased estimate for the quality of the responses.
