  Large language models (LLMs) have demonstrated unparalleled prowess in
mimicking human-like text generation and processing. Among the myriad of
applications that benefit from LLMs, automated code generation is increasingly
promising. The potential to transform natural language prompts into executable
code promises a major shift in software development practices and paves the way
for significant reductions in manual coding efforts and the likelihood of
human-induced errors. This paper reports the results of a study that evaluates
the performance of various LLMs, such as Bard, ChatGPT-3.5, ChatGPT-4, and
Claude-2, in generating Python for coding problems. We focus on how levels of
prompt specificity impact the accuracy, time efficiency, and space efficiency
of the generated code. A benchmark of 104 coding problems, each with four types
of prompts with varying degrees of tests and specificity, was employed to
examine these aspects comprehensively. Our results indicate significant
variations in performance across different LLMs and prompt types, and its key
contribution is to reveal the ideal prompting strategy for creating accurate
Python functions. This study lays the groundwork for further research in LLM
capabilities and suggests practical implications for utilizing LLMs in
automated code generation tasks and test-driven development.
