  Testing plays a crucial role in the software development cycle, enabling the
detection of bugs, vulnerabilities, and other undesirable behaviors. To perform
software testing, testers need to write code snippets that execute the program
under test. Recently, researchers have recognized the potential of large
language models (LLMs) in software testing. However, there remains a lack of
fair comparisons between different LLMs in terms of test case generation
capabilities.
  In this paper, we propose TESTEVAL, a novel benchmark for test case
generation with LLMs. We collect 210 Python programs from an online programming
platform, LeetCode, and design three different tasks: overall coverage,
targeted line/branch coverage, and targeted path coverage. We further evaluate
sixteen popular LLMs, including both commercial and open-source ones, on
TESTEVAL. We find that generating test cases to cover specific program
lines/branches/paths is still challenging for current LLMs, indicating a lack
of ability to comprehend program logic and execution paths. We have
open-sourced our dataset and benchmark pipelines at
https://llm4softwaretesting.github.io to contribute and accelerate future
research on LLMs for software testing.
