  With the emergence of Large Language Models (LLMs), there has been a
significant improvement in the programming capabilities of models, attracting
growing attention from researchers. Evaluating the programming capabilities of
LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has
numerous downstream applications. In this paper, we propose CodeApex, a
bilingual benchmark dataset focusing on the programming comprehension, code
generation, and code correction abilities of LLMs. Programming comprehension
task tests LLMs on multiple-choice exam questions covering conceptual
understanding, commonsense reasoning, and multi-hop reasoning. The code
generation task evaluates LLMs through completing C++ functions based on
provided descriptions and prototypes. The code correction task asks LLMs to fix
real-world erroneous code segments with different error messages. We evaluate
12 widely used LLMs, including both general-purpose and specialized models.
GPT-4 exhibits the best programming capabilities, achieving approximate
accuracy of 69%, 54%, and 66% on the three tasks, respectively. Compared to
human performance, there is still significant room for improvement in LLM
programming. We hope that CodeApex can serve as a reference for evaluating the
coding capabilities of LLMs, further promoting their development and growth.
