  $ $Dialogue systems are evaluated depending on their type and purpose. Two
categories are often distinguished: (1) task-oriented dialogue systems (TDS),
which are typically evaluated on utility, i.e., their ability to complete a
specified task, and (2) open domain chatbots, which are evaluated on the user
experience, i.e., based on their ability to engage a person. What is the
influence of user experience on the user satisfaction rating of TDS as opposed
to, or in addition to, utility? We collect data by providing an additional
annotation layer for dialogues sampled from the ReDial dataset, a widely used
conversational recommendation dataset. Unlike prior work, we annotate the
sampled dialogues at both the turn and dialogue level on six dialogue aspects:
relevance, interestingness, understanding, task completion, efficiency, and
interest arousal. The annotations allow us to study how different dialogue
aspects influence user satisfaction. We introduce a comprehensive set of user
experience aspects derived from the annotators' open comments that can
influence users' overall impression. We find that the concept of satisfaction
varies across annotators and dialogues, and show that a relevant turn is
significant for some annotators, while for others, an interesting turn is all
they need. Our analysis indicates that the proposed user experience aspects
provide a fine-grained analysis of user satisfaction that is not captured by a
monolithic overall human rating.
