  Search and recommender systems that take the initiative to ask clarifying
questions to better understand users' information needs are receiving
increasing attention from the research community. However, to the best of our
knowledge, there is no empirical study to quantify whether and to what extent
users are willing or able to answer these questions. In this work, we conduct
an online experiment by deploying an experimental system, which interacts with
users by asking clarifying questions against a product repository. We collect
both implicit interaction behavior data and explicit feedback from users
showing that: (a) users are willing to answer a good number of clarifying
questions (11-21 on average), but not many more than that; (b) most users
answer questions until they reach the target product, but also a fraction of
them stops due to fatigue or due to receiving irrelevant questions; (c) part of
the users' answers (12-17%) are actually opposite to the description of the
target product; while (d) most of the users (66-84%) find the question-based
system helpful towards completing their tasks. Some of the findings of the
study contradict current assumptions on simulated evaluations in the field,
while they point towards improvements in the evaluation framework and can
inspire future interactive search/recommender system designs.
