  Conversational recommender systems (CRS) that are able to interact with users
in natural language often utilize recommendation dialogs which were previously
collected with the help of paired humans, where one plays the role of a seeker
and the other as a recommender. These recommendation dialogs include items and
entities that indicate the users' preferences. In order to precisely model the
seekers' preferences and respond consistently, CRS typically rely on item and
entity annotations. A recent example of such a dataset is INSPIRED, which
consists of recommendation dialogs for sociable conversational recommendation,
where items and entities were annotated using automatic keyword or pattern
matching techniques. An analysis of this dataset unfortunately revealed that
there is a substantial number of cases where items and entities were either
wrongly annotated or annotations were missing at all. This leads to the
question to what extent automatic techniques for annotations are effective.
Moreover, it is important to study impact of annotation quality on the overall
effectiveness of a CRS in terms of the quality of the system's responses. To
study these aspects, we manually fixed the annotations in INSPIRED. We then
evaluated the performance of several benchmark CRS using both versions of the
dataset. Our analyses suggest that the improved version of the dataset, i.e.,
INSPIRED2, helped increase the performance of several benchmark CRS,
emphasizing the importance of data quality both for end-to-end learning and
retrieval-based approaches to conversational recommendation. We release our
improved dataset (INSPIRED2) publicly at
https://github.com/ahtsham58/INSPIRED2.
