  It is often desirable to distill the capabilities of large language models
(LLMs) into smaller student models due to compute and memory constraints. One
way to do this for classification tasks is via dataset synthesis, which can be
accomplished by generating examples of each label from the LLM. Prior
approaches to synthesis use few-shot prompting, which relies on the LLM's
parametric knowledge to generate usable examples. However, this leads to issues
of repetition, bias towards popular entities, and stylistic differences from
human text. In this work, we propose Synthesize by Retrieval and Refinement
(SynthesizRR), which uses retrieval augmentation to introduce variety into the
dataset synthesis process: as retrieved passages vary, the LLM is seeded with
different content to generate its examples. We empirically study the synthesis
of six datasets, covering topic classification, sentiment analysis, tone
detection, and humor, requiring complex synthesis strategies. We find that
SynthesizRR greatly improves lexical and semantic diversity, similarity to
human-written text, and distillation performance, when compared to 32-shot
prompting and four prior approaches. We release our extensive codebase at
https://github.com/amazon-science/synthesizrr
