  Recent studies developed jailbreaking attacks, which construct jailbreaking
prompts to ``fool'' LLMs into responding to harmful questions. Early-stage
jailbreaking attacks require access to model internals or significant human
efforts. More advanced attacks utilize genetic algorithms for automatic and
black-box attacks. However, the random nature of genetic algorithms
significantly limits the effectiveness of these attacks. In this paper, we
propose RLbreaker, a black-box jailbreaking attack driven by deep reinforcement
learning (DRL). We model jailbreaking as a search problem and design an RL
agent to guide the search, which is more effective and has less randomness than
stochastic search, such as genetic algorithms. Specifically, we design a
customized DRL system for the jailbreaking problem, including a novel reward
function and a customized proximal policy optimization (PPO) algorithm. Through
extensive experiments, we demonstrate that RLbreaker is much more effective
than existing jailbreaking attacks against six state-of-the-art (SOTA) LLMs. We
also show that RLbreaker is robust against three SOTA defenses and its trained
agents can transfer across different LLMs. We further validate the key design
choices of RLbreaker via a comprehensive ablation study.
