  A peculiarity of conversational search systems is that they involve
mixed-initiatives such as system-generated query clarifying questions.
Evaluating those systems at a large scale on the end task of IR is very
challenging, requiring adequate datasets containing such interactions. However,
current datasets only focus on either traditional ad-hoc IR tasks or query
clarification tasks, the latter being usually seen as a reformulation task from
the initial query. The only two datasets known to us that contain both document
relevance judgments and the associated clarification interactions are Qulac and
ClariQ. Both are based on the TREC Web Track 2009-12 collection, but cover a
very limited number of topics (237 topics), far from being enough for training
and testing conversational IR models. To fill the gap, we propose a methodology
to automatically build large-scale conversational IR datasets from ad-hoc IR
datasets in order to facilitate explorations on conversational IR. Our
methodology is based on two processes: 1) generating query clarification
interactions through query clarification and answer generators, and 2)
augmenting ad-hoc IR datasets with simulated interactions. In this paper, we
focus on MsMarco and augment it with query clarification and answer
simulations. We perform a thorough evaluation showing the quality and the
relevance of the generated interactions for each initial query. This paper
shows the feasibility and utility of augmenting ad-hoc IR datasets for
conversational IR.
