  Large Language Models (LLMs) are being deployed across various domains today.
However, their capacity to solve Capture the Flag (CTF) challenges in
cybersecurity has not been thoroughly evaluated. To address this, we develop a
novel method to assess LLMs in solving CTF challenges by creating a scalable,
open-source benchmark database specifically designed for these applications.
This database includes metadata for LLM testing and adaptive learning,
compiling a diverse range of CTF challenges from popular competitions.
Utilizing the advanced function calling capabilities of LLMs, we build a fully
automated system with an enhanced workflow and support for external tool calls.
Our benchmark dataset and automated framework allow us to evaluate the
performance of five LLMs, encompassing both black-box and open-source models.
This work lays the foundation for future research into improving the efficiency
of LLMs in interactive cybersecurity tasks and automated task planning. By
providing a specialized dataset, our project offers an ideal platform for
developing, testing, and refining LLM-based approaches to vulnerability
detection and resolution. Evaluating LLMs on these challenges and comparing
with human performance yields insights into their potential for AI-driven
cybersecurity solutions to perform real-world threat management. We make our
dataset open source to public https://github.com/NYU-LLM-CTF/LLM_CTF_Database
along with our playground automated framework
https://github.com/NYU-LLM-CTF/llm_ctf_automation.
