  Large Visual Language Models (VLMs) such as GPT-4 have achieved remarkable
success in generating comprehensive and nuanced responses, surpassing the
capabilities of large language models. However, with the integration of visual
inputs, new security concerns emerge, as malicious attackers can exploit
multiple modalities to achieve their objectives. This has led to increasing
attention on the vulnerabilities of VLMs to jailbreak. Most existing research
focuses on generating adversarial images or nonsensical image collections to
compromise these models. However, the challenge of leveraging meaningful images
to produce targeted textual content using the VLMs' logical comprehension of
images remains unexplored. In this paper, we explore the problem of logical
jailbreak from meaningful images to text. To investigate this issue, we
introduce a novel dataset designed to evaluate flowchart image jailbreak.
Furthermore, we develop a framework for text-to-text jailbreak using VLMs.
Finally, we conduct an extensive evaluation of the framework on GPT-4o and
GPT-4-vision-preview, with jailbreak rates of 92.8% and 70.0%, respectively.
Our research reveals significant vulnerabilities in current VLMs concerning
image-to-text jailbreak. These findings underscore the need for a deeper
examination of the security flaws in VLMs before their practical deployment.
