  The recent development of Sora leads to a new era in text-to-video (T2V)
generation. Along with this comes the rising concern about its security risks.
The generated videos may contain illegal or unethical content, and there is a
lack of comprehensive quantitative understanding of their safety, posing a
challenge to their reliability and practical deployment. Previous evaluations
primarily focus on the quality of video generation. While some evaluations of
text-to-image models have considered safety, they cover fewer aspects and do
not address the unique temporal risk inherent in video generation. To bridge
this research gap, we introduce T2VSafetyBench, a new benchmark designed for
conducting safety-critical assessments of text-to-video models. We define 12
critical aspects of video generation safety and construct a malicious prompt
dataset using LLMs and jailbreaking prompt attacks. Based on our evaluation
results, we draw several important findings, including: 1) no single model
excels in all aspects, with different models showing various strengths; 2) the
correlation between GPT-4 assessments and manual reviews is generally high; 3)
there is a trade-off between the usability and safety of text-to-video
generative models. This indicates that as the field of video generation rapidly
advances, safety risks are set to surge, highlighting the urgency of
prioritizing video safety. We hope that T2VSafetyBench can provide insights for
better understanding the safety of video generation in the era of generative
AI.
