  While there has been a recent burgeoning of applications at the intersection
of natural and programming languages, such as code generation and code
summarization, these applications are usually English-centric. This creates a
barrier for program developers who are not proficient in English. To mitigate
this gap in technology development across languages, we propose a multilingual
dataset, MCoNaLa, to benchmark code generation from natural language commands
extending beyond English. Modeled off of the methodology from the English
Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896
NL-code pairs in three languages: Spanish, Japanese, and Russian. We present a
quantitative evaluation of performance on the MCoNaLa dataset by testing with
state-of-the-art code generation systems. While the difficulties vary across
these three languages, all systems lag significantly behind their English
counterparts, revealing the challenges in adapting code generation to new
languages.
