  As large language models (LLMs) become increasingly prevalent across many
real-world applications, understanding and enhancing their robustness to
adversarial attacks is of paramount importance. Existing methods for
identifying adversarial prompts tend to focus on specific domains, lack
diversity, or require extensive human annotations. To address these
limitations, we present Rainbow Teaming, a novel black-box approach for
producing a diverse collection of adversarial prompts. Rainbow Teaming casts
adversarial prompt generation as a quality-diversity problem, and uses
open-ended search to generate prompts that are both effective and diverse.
Focusing on the safety domain, we use Rainbow Teaming to target various
state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach
reveals hundreds of effective adversarial prompts, with an attack success rate
exceeding 90% across all tested models. Furthermore, we demonstrate that
fine-tuning models with synthetic data generated by the Rainbow Teaming method
significantly enhances their safety without sacrificing general performance or
helpfulness. We additionally explore the versatility of Rainbow Teaming by
applying it to question answering and cybersecurity, showcasing its potential
to drive robust open-ended self-improvement in a wide range of applications.
