  The security concerns surrounding Large Language Models (LLMs) have been
extensively explored, yet the safety of Multimodal Large Language Models
(MLLMs) remains understudied. In this paper, we observe that Multimodal Large
Language Models (MLLMs) can be easily compromised by query-relevant images, as
if the text query itself were malicious. To address this, we introduce
MM-SafetyBench, a comprehensive framework designed for conducting
safety-critical evaluations of MLLMs against such image-based manipulations. We
have compiled a dataset comprising 13 scenarios, resulting in a total of 5,040
text-image pairs. Our analysis across 12 state-of-the-art models reveals that
MLLMs are susceptible to breaches instigated by our approach, even when the
equipped LLMs have been safety-aligned. In response, we propose a
straightforward yet effective prompting strategy to enhance the resilience of
MLLMs against these types of attacks. Our work underscores the need for a
concerted effort to strengthen and enhance the safety measures of open-source
MLLMs against potential malicious exploits. The resource is available at
https://github.com/isXinLiu/MM-SafetyBench
