  Recent explorations with commercial Large Language Models (LLMs) have shown
that non-expert users can jailbreak LLMs by simply manipulating their prompts;
resulting in degenerate output behavior, privacy and security breaches,
offensive outputs, and violations of content regulator policies. Limited
studies have been conducted to formalize and analyze these attacks and their
mitigations. We bridge this gap by proposing a formalism and a taxonomy of
known (and possible) jailbreaks. We survey existing jailbreak methods and their
effectiveness on open-source and commercial LLMs (such as GPT-based models,
OPT, BLOOM, and FLAN-T5-XXL). We further discuss the challenges of jailbreak
detection in terms of their effectiveness against known attacks. For further
analysis, we release a dataset of model outputs across 3700 jailbreak prompts
over 4 tasks.
