  Large Language Models (LLMs) have shown great potential in code generation.
However, current LLMs still cannot reliably generate correct code. Moreover, it
is unclear what kinds of code generation errors LLMs can make. To address this,
we conducted an empirical study to analyze incorrect code snippets generated by
six popular LLMs on the HumanEval dataset. We analyzed these errors alongside
two dimensions of error characteristics -- semantic characteristics and
syntactic characteristics -- to derive a comprehensive code generation error
taxonomy for LLMs through open coding and thematic analysis. We then labeled
all 557 incorrect code snippets based on this taxonomy. Our results showed that
the six LLMs exhibited similar distributions of syntactic characteristics while
different distributions of semantic characteristics. Furthermore, we analyzed
the correlation between different error characteristics and factors such as
task complexity, code length, and test-pass rate. Finally, we highlight the
challenges that LLMs may encounter when generating code and propose
implications for future research on reliable code generation with LLMs.
