  State-of-the-art dialogue models still often stumble with regards to factual
accuracy and self-contradiction. Anecdotally, they have been observed to fail
to maintain character identity throughout discourse; and more specifically, may
take on the role of their interlocutor. In this work we formalize and quantify
this deficiency, and show experimentally through human evaluations that this is
indeed a problem. In contrast, we show that discriminative models trained
specifically to recognize who is speaking can perform well; and further, these
can be used as automated metrics. Finally, we evaluate a wide variety of
mitigation methods, including changes to model architecture, training protocol,
and decoding strategy. Our best models reduce mistaken identity issues by
nearly 65% according to human annotators, while simultaneously improving
engagingness. Despite these results, we find that maintaining character
identity still remains a challenging problem.
