  Large language models (LLMs) have made significant progress in generating
codes from textual prompts. However, existing benchmarks have mainly
concentrated on translating English prompts to multilingual codes or have been
constrained to very limited natural languages (NLs). These benchmarks have
overlooked the vast landscape of massively multilingual NL to multilingual
code, leaving a critical gap in the evaluation of multilingual LLMs. In
response, we introduce HumanEval-XL, a massively multilingual code generation
benchmark specifically crafted to address this deficiency. HumanEval-XL
establishes connections between 23 NLs and 12 programming languages (PLs), and
comprises of a collection of 22,080 prompts with an average of 8.33 test cases.
By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a
comprehensive evaluation platform for multilingual LLMs, allowing the
assessment of the understanding of different NLs. Our work serves as a
pioneering step towards filling the void in evaluating NL generalization in the
area of multilingual code generation. We make our evaluation code and data
publicly available at \url{https://github.com/FloatAI/humaneval-xl}.
