  Large Code Generation Models (LCGMs) have garnered significant attention and
achieved promising results across various programming tasks. However, concerns
arise regarding performance when using non-English prompts, as these models are
primarily trained on English-centric corpora, and most programming language
tokens resemble English. Existing benchmarks often rely on English programming
questions and limited manual unit test cases, inadequately assessing
LCGM-generated code quality. This paper investigates code quality differences,
specifically effectiveness and efficiency, when employing different natural
languages as inputs, focusing on Chinese and English due to their prominent
corpora and LCGM availability. Evaluating LCGM-generated code quality under
bilingual inputs presents three challenges: (1) lack of high-quality bilingual
programming question datasets, (2) insufficient unit test cases for
comprehensive correctness verification, and (3) limited support for comparing
generated code performance. To address these challenges, we curated a test
suite of 52 bilingual programming questions and developed automated input
generators for each. We enhanced correctness verification by sampling larger
unit test cases and estimated code performance by profiling execution time
relative to input size growth. Using this framework, we conducted an empirical
study on six state-of-the-art LCGMs. The results revealed that LCGM-generated
code exhibits varying bilingual correctness on an average of 10.5% of tasks,
with 39.5% of correct code showing diverse bilingual performance differences.
Our findings suggested LCGMs may not consistently generate high-quality code
across different languages, providing insights for future research directions.
