  Existing efforts in safeguarding LLMs are limited in actively exposing the
vulnerabilities of the target LLM and readily adapting to newly emerging safety
risks. To address this, we present Purple-teaming LLMs with Adversarial
Defender training (PAD), a pipeline designed to safeguard LLMs by novelly
incorporating the red-teaming (attack) and blue-teaming (safety training)
techniques. In PAD, we automatically collect conversational data that cover the
vulnerabilities of an LLM around specific safety risks in a self-play manner,
where the attacker aims to elicit unsafe responses and the defender generates
safe responses to these attacks. We then update both modules in a generative
adversarial network style by training the attacker to elicit more unsafe
responses and updating the defender to identify them and explain the unsafe
reason. Experimental results demonstrate that PAD significantly outperforms
existing baselines in both finding effective attacks and establishing a robust
safe guardrail. Furthermore, our findings indicate that PAD excels in striking
a balance between safety and overall model quality. We also reveal key
challenges in safeguarding LLMs, including defending multi-turn attacks and the
need for more delicate strategies to identify specific risks.
