  Collecting high quality conversational data can be very expensive for most
applications and infeasible for others due to privacy, ethical, or similar
concerns. A promising direction to tackle this problem is to generate synthetic
dialogues by prompting large language models. In this work, we use a small set
of expert-written conversations as in-context examples to synthesize a social
conversation dataset using prompting. We perform several thorough evaluations
of our synthetic conversations compared to human-collected conversations. This
includes various dimensions of conversation quality with human evaluation
directly on the synthesized conversations, and interactive human evaluation of
chatbots fine-tuned on the synthetically generated dataset. We additionally
demonstrate that this prompting approach is generalizable to multi-party
conversations, providing potential to create new synthetic data for multi-party
tasks. Our synthetic multi-party conversations were rated more favorably across
all measured dimensions compared to conversation excerpts sampled from a
human-collected multi-party dataset.
