  Security vulnerabilities in modern software are prevalent and harmful. While
automated vulnerability detection tools have made promising progress, their
scalability and applicability remain challenging. Recently, Large Language
Models (LLMs), such as GPT-4 and CodeLlama, have demonstrated remarkable
performance on code-related tasks. However, it is unknown whether such LLMs can
do complex reasoning over code. In this work, we explore whether pre-trained
LLMs can detect security vulnerabilities and address the limitations of
existing tools. We evaluate the effectiveness of pre-trained LLMs, in terms of
performance, explainability, and robustness, on a set of five diverse security
benchmarks spanning two languages, Java and C/C++, and covering both synthetic
and real-world projects.
  Overall, all LLMs show modest effectiveness in end-to-end reasoning about
vulnerabilities, obtaining an average of 60% accuracy across all datasets.
However, we observe that LLMs show promising abilities at performing parts of
the analysis correctly, such as identifying vulnerability-related
specifications (e.g., sources and sinks) and leveraging natural language
information to understand code behavior (e.g., to check if code is sanitized).
Further, LLMs are relatively much better at detecting simpler vulnerabilities
that typically only need local reasoning (e.g., Integer Overflows and NULL
pointer dereference). We find that advanced prompting strategies that involve
step-by-step analysis significantly improve performance of LLMs on real-world
datasets (improving F1 score by up to 0.25 on average). Finally, we share our
insights and recommendations for future work on leveraging LLMs for
vulnerability detection.
