  We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3,
to continue the conversation on empirically measuring LLM cybersecurity risks
and capabilities. CYBERSECEVAL 3 assesses 8 different risks across two broad
categories: risk to third parties, and risk to application developers and end
users. Compared to previous work, we add new areas focused on offensive
security capabilities: automated social engineering, scaling manual offensive
cyber operations, and autonomous offensive cyber operations. In this paper we
discuss applying these benchmarks to the Llama 3 models and a suite of
contemporaneous state-of-the-art LLMs, enabling us to contextualize risks both
with and without mitigations in place.
