  In Conversational Recommendation Systems (CRS), a user can provide feedback
on recommended items at each interaction turn, leading the CRS towards more
desirable recommendations. Currently, different types of CRS offer various
possibilities for feedback, i.e., natural language feedback, or answering
clarifying questions. In most cases, a user simulator is employed for training
as well as evaluating the CRS. Such user simulators typically critique the
current retrieved items based on knowledge of a single target item. Still,
evaluating systems in offline settings with simulators suffers from problems,
such as focusing entirely on a single target item (not addressing the
exploratory nature of a recommender system), and exhibiting extreme patience
(consistent feedback over a large number of turns). To overcome these
limitations, we obtain extra judgements for a selection of alternative items in
common CRS datasets, namely Shoes and Fashion IQ Dresses. Going further, we
propose improved user simulators that allow simulated users not only to express
their preferences about alternative items to their original target, but also to
change their mind and level of patience. In our experiments using the relative
image captioning CRS setting and different CRS models, we find that using the
knowledge of alternatives by the simulator can have a considerable impact on
the evaluation of existing CRS models, specifically that the existing
single-target evaluation underestimates their effectiveness, and when simulated
users are allowed to instead consider alternatives, the system can rapidly
respond to more quickly satisfy the user.
