  Unit testing is an essential activity in software development for verifying
the correctness of software components. However, manually writing unit tests is
challenging and time-consuming. The emergence of Large Language Models (LLMs)
offers a new direction for automating unit test generation. Existing research
primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed
prompting strategies, leaving the capabilities of advanced open-source LLMs
with various prompting settings unexplored. Particularly, open-source LLMs
offer advantages in data privacy protection and have demonstrated superior
performance in some tasks. Moreover, effective prompting is crucial for
maximizing LLMs' capabilities. In this paper, we conduct the first empirical
study to fill this gap, based on 17 Java projects, five widely-used open-source
LLMs with different structures and parameter sizes, and comprehensive
evaluation metrics. Our findings highlight the significant influence of various
prompt factors, show the performance of open-source LLMs compared to the
commercial GPT-4 and the traditional Evosuite, and identify limitations in
LLM-based unit test generation. We then derive a series of implications from
our study to guide future research and practical use of LLM-based unit test
generation.
