  We systematically study the quality of 4,066 ChatGPT-generated code
implemented in two popular programming languages, i.e., Java and Python, for
2,033 programming tasks. The goal of this work is three folds. First, we
analyze the correctness of ChatGPT on code generation tasks and uncover the
factors that influence its effectiveness, including task difficulty,
programming language, time that tasks are introduced, and program size. Second,
we identify and characterize potential issues with the quality of
ChatGPT-generated code. Last, we provide insights into how these issues can be
mitigated. Experiments highlight that out of 4,066 programs generated by
ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong
outputs, and 177 programs contain compilation or runtime errors. Additionally,
we further analyze other characteristics of the generated code through static
analysis tools, such as code style and maintainability, and find that 1,930
ChatGPT-generated code snippets suffer from maintainability issues.
Subsequently, we investigate ChatGPT's self-repairing ability and its
interaction with static analysis tools to fix the errors uncovered in the
previous step. Experiments suggest that ChatGPT can partially address these
challenges, improving code quality by more than 20%, but there are still
limitations and opportunities for improvement. Overall, our study provides
valuable insights into the current limitations of ChatGPT and offers a roadmap
for future research and development efforts to enhance the code generation
capabilities of AI models like ChatGPT.
