  Code generation aims to synthesize code and fulfill functional requirements
based on natural language (NL) specifications, which can greatly improve
development efficiency. In the era of large language models (LLMs), large code
models (LCMs) have been recently proposed to generate source code. LCMs can
generate highly feasible solutions for programming problems described in
natural language. Despite the effectiveness, we observe a noticeable
multilingual bias in the generation performance of LCMs. Specifically, LCMs
demonstrate proficiency in generating solutions when provided with instructions
in English, yet may falter when faced with semantically equivalent instructions
in other NLs such as Chinese. Moreover, the ability of LCMs to generate code
exhibits variety across different programming languages (PLs), such as Python
and C++. The observed phenomenon indicates the presence of multi-lingual bias
within the generative capabilities of LCMs, which has remained unexplored.
  In this paper, we aim to investigate the multi-lingual bias that exists in
current LCMs. First, we initiate our investigation by constructing the first
multi-lingual evaluation benchmark X-HumanEval-X, enabling us to systematically
evaluate the extent of multi-lingual bias that exists in current LCMs. In our
large-scale experiments on nine popular LCMs, we observe a pronounced
multi-lingual bias of LCMs in code generation, including multi-NL and multi-PL
bias. Specifically, when using Chinese instructions, the code generation
capabilities of LCMs decrease by at least 13% in terms of the Pass@1 metric.
Furthermore, LCMs perform variously across different programming languages,
e.g., the performance gap between Python and C++ reaches as high as 20.9%. ...
