  One challenge for evaluating current sequence- or dialogue-level chatbots,
such as Empathetic Open-domain Conversation Models, is to determine whether the
chatbot performs in an emotionally consistent way. The most recent work only
evaluates on the aspects of context coherence, language fluency, response
diversity, or logical self-consistency between dialogues. This work proposes
training an evaluator to determine the emotional consistency of chatbots.
