  Large Language Models (LLMs) have demonstrated remarkable performance on
assisting humans in programming and facilitating programming automation.
However, existing benchmarks for evaluating the code understanding and
generation capacities of LLMs suffer from severe limitations. First, most
benchmarks are insufficient as they focus on a narrow range of popular
programming languages and specific tasks, whereas real-world software
development scenarios show a critical need to implement systems with
multilingual and multitask programming environments to satisfy diverse
requirements. Second, most benchmarks fail to consider the actual executability
and the consistency of execution results of the generated code. To bridge these
gaps between existing benchmarks and expectations from practical applications,
we introduce CodeScope, an execution-based, multilingual, multitask,
multidimensional evaluation benchmark for comprehensively measuring LLM
capabilities on coding tasks. CodeScope covers 43 programming languages and
eight coding tasks. It evaluates the coding performance of LLMs from three
dimensions (perspectives): length, difficulty, and efficiency. To facilitate
execution-based evaluations of code generation, we develop MultiCodeEngine, an
automated code execution engine that supports 14 programming languages.
Finally, we systematically evaluate and analyze eight mainstream LLMs and
demonstrate the superior breadth and challenges of CodeScope for evaluating
LLMs on code understanding and generation tasks compared to other benchmarks.
The CodeScope benchmark and code are publicly available at
https://github.com/WeixiangYAN/CodeScope.
