  Conversational question-answering (CQA) systems aim to create interactive
search systems that effectively retrieve information by interacting with users.
To replicate human-to-human conversations, existing work uses human annotators
to play the roles of the questioner (student) and the answerer (teacher).
Despite its effectiveness, challenges exist as human annotation is
time-consuming, inconsistent, and not scalable. To address this issue and
investigate the applicability of large language models (LLMs) in CQA
simulation, we propose a simulation framework that employs zero-shot learner
LLMs for simulating teacher-student interactions. Our framework involves two
LLMs interacting on a specific topic, with the first LLM acting as a student,
generating questions to explore a given search topic. The second LLM plays the
role of a teacher by answering questions and is equipped with additional
information, including a text on the given topic. We implement both the student
and teacher by zero-shot prompting the GPT-4 model. To assess the effectiveness
of LLMs in simulating CQA interactions and understand the disparities between
LLM- and human-generated conversations, we evaluate the simulated data from
various perspectives. We begin by evaluating the teacher's performance through
both automatic and human assessment. Next, we evaluate the performance of the
student, analyzing and comparing the disparities between questions generated by
the LLM and those generated by humans. Furthermore, we conduct extensive
analyses to thoroughly examine the LLM performance by benchmarking
state-of-the-art reading comprehension models on both datasets. Our results
reveal that the teacher LLM generates lengthier answers that tend to be more
accurate and complete. The student LLM generates more diverse questions,
covering more aspects of a given topic.
