  There is a multitude of novel generative models for open-domain
conversational systems; however, there is no systematic evaluation of different
systems. Systematic comparisons require consistency in experimental design,
evaluation sets, conversational systems and their outputs, and statistical
analysis. We lay out a protocol for the evaluation of conversational models
using head-to-head pairwise comparison. We analyze ten recent models that claim
state-of-the-art performance using a paired head-to-head performance
(win-loss-tie) on five evaluation datasets. Our findings show that DialoGPT and
Blender are superior systems using Bradley-Terry model and TrueSkill ranking
methods. These findings demonstrate the feasibility of our protocol to evaluate
conversational agents and evaluation sets. Finally, we make all code and
evaluations publicly available for researchers to compare their model to other
state-of-the-art dialog models.
