  Spurred by recent advances in Large Language Models (LLMs), virtual
assistants are poised to take a leap forward in terms of their dialogue
capabilities. Yet a major bottleneck to achieving genuinely transformative
task-oriented dialogue capabilities remains the scarcity of high quality data.
Existing datasets, while impressive in scale, have limited domain coverage and
contain few genuinely challenging conversational phenomena; those which are
present are typically unlabelled, making it difficult to assess the strengths
and weaknesses of models without time-consuming and costly human evaluation.
Moreover, creating high quality dialogue data has until now required
considerable human input, limiting both the scale of these datasets and the
ability to rapidly bootstrap data for a new target domain. We aim to overcome
these issues with LUCID, a modularised and highly automated LLM-driven data
generation system that produces realistic, diverse and challenging dialogues.
We use LUCID to generate a seed dataset of 4,277 conversations across 100
intents to demonstrate its capabilities, with a human review finding
consistently high quality labels in the generated data.
