  With the rapid development of Large Language Models (LLMs), increasing
attention has been paid to their safety concerns. Consequently, evaluating the
safety of LLMs has become an essential task for facilitating the broad
applications of LLMs. Nevertheless, the absence of comprehensive safety
evaluation benchmarks poses a significant impediment to effectively assess and
enhance the safety of LLMs. In this work, we present SafetyBench, a
comprehensive benchmark for evaluating the safety of LLMs, which comprises
11,435 diverse multiple choice questions spanning across 7 distinct categories
of safety concerns. Notably, SafetyBench also incorporates both Chinese and
English data, facilitating the evaluation in both languages. Our extensive
tests over 25 popular Chinese and English LLMs in both zero-shot and few-shot
settings reveal a substantial performance advantage for GPT-4 over its
counterparts, and there is still significant room for improving the safety of
current LLMs. We also demonstrate that the measured safety understanding
abilities in SafetyBench are correlated with safety generation abilities. Data
and evaluation guidelines are available at
\url{https://github.com/thu-coai/SafetyBench}{https://github.com/thu-coai/SafetyBench}.
Submission entrance and leaderboard are available at
\url{https://llmbench.ai/safety}{https://llmbench.ai/safety}.
