  Unit testing, crucial for identifying bugs in code modules like classes and
methods, is often neglected by developers due to time constraints. Automated
test generation techniques have emerged to address this, but often lack
readability and require developer intervention. Large Language Models (LLMs),
like GPT and Mistral, show promise in software engineering, including in test
generation. However, their effectiveness remains unclear.
  This study conducts the first comprehensive investigation of LLMs, evaluating
the effectiveness of four LLMs and five prompt engineering techniques, for unit
test generation. We analyze 216\,300 tests generated by the selected advanced
instruct-tuned LLMs for 690 Java classes collected from diverse datasets. We
assess correctness, understandability, coverage, and bug detection capabilities
of LLM-generated tests, comparing them to EvoSuite, a popular automated testing
tool. While LLMs show potential, improvements in test correctness are
necessary. This study reveals the strengths and limitations of LLMs compared to
traditional methods, paving the way for further research on LLMs in software
engineering.
