  Conversational recommender systems (CRS) enable users to articulate their
preferences and provide feedback through natural language. With the advent of
large language models (LLMs), the potential to enhance user engagement with CRS
and augment the recommendation process with LLM-generated content has received
increasing attention. However, the efficacy of LLM-powered CRS is contingent
upon the use of prompts, and the subjective perception of recommendation
quality can differ across various recommendation domains. Therefore, we have
developed a ChatGPT-based CRS to investigate the impact of these two factors,
prompt guidance (PG) and recommendation domain (RD), on the overall user
experience of the system. We conducted an online empirical study (N = 100) by
employing a mixed-method approach that utilized a between-subjects design for
the variable of PG (with vs. without) and a within-subjects design for RD (book
recommendations vs. job recommendations). The findings reveal that PG can
substantially enhance the system's explainability, adaptability, perceived ease
of use, and transparency. Moreover, users are inclined to perceive a greater
sense of novelty and demonstrate a higher propensity to engage with and try
recommended items in the context of book recommendations as opposed to job
recommendations. Furthermore, the influence of PG on certain user experience
metrics and interactive behaviors appears to be modulated by the recommendation
domain, as evidenced by the interaction effects between the two examined
factors. This work contributes to the user-centered evaluation of ChatGPT-based
CRS by investigating two prominent factors and offers practical design
guidance.
