  As Artificial General Intelligence (AGI) becomes increasingly integrated into
various facets of human life, ensuring the safety and ethical alignment of such
systems is paramount. Previous studies primarily focus on single-modality
threats, which may not suffice given the integrated and complex nature of
cross-modality interactions. We introduce a novel safety alignment challenge
called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety
alignment. Specifically, it considers cases where single modalities are safe
independently but could potentially lead to unsafe or unethical outputs when
combined. To empirically investigate this problem, we developed the SIUO, a
cross-modality benchmark encompassing 9 critical safety domains, such as
self-harm, illegal activities, and privacy violations. Our findings reveal
substantial safety vulnerabilities in both closed- and open-source LVLMs, such
as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably
interpret and respond to complex, real-world scenarios.
