  Recent advancements in large language models (LLMs) have demonstrated
exceptional success in a wide range of general domain tasks, such as question
answering and following instructions. Moreover, LLMs have shown potential in
various software engineering applications. In this study, we present a
systematic comparison of test suites generated by the ChatGPT LLM and the
state-of-the-art SBST tool EvoSuite. Our comparison is based on several
critical factors, including correctness, readability, code coverage, and bug
detection capability. By highlighting the strengths and weaknesses of LLMs
(specifically ChatGPT) in generating unit test cases compared to EvoSuite, this
work provides valuable insights into the performance of LLMs in solving
software engineering problems. Overall, our findings underscore the potential
of LLMs in software engineering and pave the way for further research in this
area.
