  In an information-seeking conversation, a user may ask questions that are
under-specified or unanswerable. An ideal agent would interact by initiating
different response types according to the available knowledge sources. However,
most current studies either fail to or artificially incorporate such agent-side
initiative. This work presents InSCIt, a dataset for Information-Seeking
Conversations with mixed-initiative Interactions. It contains 4.7K user-agent
turns from 805 human-human conversations where the agent searches over
Wikipedia and either directly answers, asks for clarification, or provides
relevant information to address user queries. The data supports two subtasks,
evidence passage identification and response generation, as well as a human
evaluation protocol to assess model performance. We report results of two
systems based on state-of-the-art models of conversational knowledge
identification and open-domain question answering. Both systems significantly
underperform humans, suggesting ample room for improvement in future studies.
