  The increasing development of large language models (LLMs) in code generation
has drawn significant attention among researchers. To enhance LLM-based code
generation ability, current efforts are predominantly directed towards
collecting high-quality datasets and leveraging diverse training technologies.
However, there is a notable lack of comprehensive studies examining the
limitations and boundaries of these existing methods. To bridge this gap, we
conducted an extensive empirical study evaluating the performance of three
leading closed-source LLMs and four popular open-source LLMs on three commonly
used benchmarks. Our investigation, which evaluated the length, cyclomatic
complexity and API number of the generated code, revealed that these LLMs face
challenges in generating successful code for more complex problems, and tend to
produce code that is shorter yet more complicated as compared to canonical
solutions. Additionally, we developed a taxonomy of bugs for incorrect codes
that includes three categories and 12 sub-categories, and analyze the root
cause for common bug types. Furthermore, to better understand the performance
of LLMs in real-world projects, we manually created a real-world benchmark
comprising 140 code generation tasks. Our analysis highlights distinct
differences in bug distributions between actual scenarios and existing
benchmarks. Finally, we propose a novel training-free iterative method that
introduces self-critique, enabling LLMs to critique and correct their generated
code based on bug types and compiler feedback. Experimental results demonstrate
that our approach can significantly mitigate bugs and increase the passing rate
by 29.2% after two iterations, indicating substantial potential for LLMs to
handle more complex problems.
