  In many text-generation problems, users may prefer not only a single
response, but a diverse range of high-quality outputs from which to choose.
Quality-diversity (QD) search algorithms aim at such outcomes, by continually
improving and diversifying a population of candidates. However, the
applicability of QD to qualitative domains, like creative writing, has been
limited by the difficulty of algorithmically specifying measures of quality and
diversity. Interestingly, recent developments in language models (LMs) have
enabled guiding search through AI feedback, wherein LMs are prompted in natural
language to evaluate qualitative aspects of text. Leveraging this development,
we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an
evolutionary algorithm applies LMs to both generate variation and evaluate the
quality and diversity of candidate text. When assessed on creative writing
domains, QDAIF covers more of a specified search space with high-quality
samples than do non-QD controls. Further, human evaluation of QDAIF-generated
creative texts validates reasonable agreement between AI and human evaluation.
Our results thus highlight the potential of AI feedback to guide open-ended
search for creative and original solutions, providing a recipe that seemingly
generalizes to many domains and modalities. In this way, QDAIF is a step
towards AI systems that can independently search, diversify, evaluate, and
improve, which are among the core skills underlying human society's capacity
for innovation.
