  Jailbreak attacks in large language models (LLMs) entail inducing the models
to generate content that breaches ethical and legal norm through the use of
malicious prompts, posing a substantial threat to LLM security. Current
strategies for jailbreak attack and defense often focus on optimizing locally
within specific algorithmic frameworks, resulting in ineffective optimization
and limited scalability. In this paper, we present a systematic analysis of the
dependency relationships in jailbreak attack and defense techniques,
generalizing them to all possible attack surfaces. We employ directed acyclic
graphs (DAGs) to position and analyze existing jailbreak attacks, defenses, and
evaluation methodologies, and propose three comprehensive, automated, and
logical frameworks. \texttt{AutoAttack} investigates dependencies in two lines
of jailbreak optimization strategies: genetic algorithm (GA)-based attacks and
adversarial-generation-based attacks, respectively. We then introduce an
ensemble jailbreak attack to exploit these dependencies. \texttt{AutoDefense}
offers a mixture-of-defenders approach by leveraging the dependency
relationships in pre-generative and post-generative defense strategies.
\texttt{AutoEvaluation} introduces a novel evaluation method that distinguishes
hallucinations, which are often overlooked, from jailbreak attack and defense
responses. Through extensive experiments, we demonstrate that the proposed
ensemble jailbreak attack and defense framework significantly outperforms
existing research.
