  The predominant approach to open-domain dialog generation relies on
end-to-end training of neural models on chat datasets. However, this approach
provides little insight as to what these models learn (or do not learn) about
engaging in dialog. In this study, we analyze the internal representations
learned by neural open-domain dialog systems and evaluate the quality of these
representations for learning basic conversational skills. Our results suggest
that standard open-domain dialog systems struggle with answering questions,
inferring contradiction, and determining the topic of conversation, among other
tasks. We also find that the dyadic, turn-taking nature of dialog is not fully
leveraged by these models. By exploring these limitations, we highlight the
need for additional research into architectures and training methods that can
better capture high-level information about dialog.
