  Large Language Models (LLMs) and Multi-Modal LLMs (MLLMs) have played a
critical role in numerous applications. However, current LLMs are vulnerable to
prompt-based attacks, with jailbreaking attacks enabling LLMs to generate
harmful content, while hijacking attacks manipulate the model to perform
unintended tasks, underscoring the necessity for detection methods.
Unfortunately, existing detecting approaches are usually tailored to specific
attacks, resulting in poor generalization in detecting various attacks across
different modalities. To address it, we propose JailGuard, a universal
detection framework for jailbreaking and hijacking attacks across LLMs and
MLLMs. JailGuard operates on the principle that attacks are inherently less
robust than benign ones, regardless of method or modality. Specifically,
JailGuard mutates untrusted inputs to generate variants and leverages the
discrepancy of the variants' responses on the model to distinguish attack
samples from benign samples. We implement 18 mutators for text and image inputs
and design a mutator combination policy to further improve detection
generalization. To evaluate the effectiveness of JailGuard, we build the first
comprehensive multi-modal attack dataset, containing 11,000 data items across
15 known attack types. The evaluation suggests that JailGuard achieves the best
detection accuracy of 86.14%/82.90% on text and image inputs, outperforming
state-of-the-art methods by 11.81%-25.73% and 12.20%-21.40%.
