  Despite advancements in on-topic dialogue systems, effectively managing topic
shifts within dialogues remains a persistent challenge, largely attributed to
the limited availability of training datasets. To address this issue, we
propose Multi-Passage to Dialogue (MP2D), a data generation framework that
automatically creates conversational question-answering datasets with natural
topic transitions. By leveraging the relationships between entities in a
knowledge graph, MP2D maps the flow of topics within a dialogue, effectively
mirroring the dynamics of human conversation. It retrieves relevant passages
corresponding to the topics and transforms them into dialogues through the
passage-to-dialogue method. Through quantitative and qualitative experiments,
we demonstrate MP2D's efficacy in generating dialogue with natural topic
shifts. Furthermore, this study introduces a novel benchmark for topic shift
dialogues, TS-WikiDialog. Utilizing the dataset, we demonstrate that even Large
Language Models (LLMs) struggle to handle topic shifts in dialogue effectively,
and we showcase the performance improvements of models trained on datasets
generated by MP2D across diverse topic shift dialogue tasks.
