  Jailbreak attacks aim to induce Large Language Models (LLMs) to generate
harmful responses for forbidden instructions, presenting severe misuse threats
to LLMs. Up to now, research into jailbreak attacks and defenses is emerging,
however, there is (surprisingly) no consensus on how to evaluate whether a
jailbreak attempt is successful. In other words, the methods to assess the
harmfulness of an LLM's response are varied, such as manual annotation or
prompting GPT-4 in specific ways. Each approach has its own set of strengths
and weaknesses, impacting their alignment with human values, as well as the
time and financial cost. This diversity in evaluation presents challenges for
researchers in choosing suitable evaluation methods and conducting fair
comparisons across different jailbreak attacks and defenses. In this paper, we
conduct a comprehensive analysis of jailbreak evaluation methodologies, drawing
from nearly ninety jailbreak research released between May 2023 and April 2024.
Our study introduces a systematic taxonomy of jailbreak evaluators, offering
in-depth insights into their strengths and weaknesses, along with the current
status of their adaptation. Moreover, to facilitate subsequent research, we
propose JailbreakEval, a user-friendly toolkit focusing on the evaluation of
jailbreak attempts. It includes various well-known evaluators out-of-the-box,
so that users can obtain evaluation results with only a single command.
JailbreakEval also allows users to customize their own evaluation workflow in a
unified framework with the ease of development and comparison. In summary, we
regard JailbreakEval to be a catalyst that simplifies the evaluation process in
jailbreak research and fosters an inclusive standard for jailbreak evaluation
within the community.
