  To extend the scope of coding queries to more realistic settings, we propose
ODEX, the first Open-Domain EXecution-based natural language (NL) to Python
code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse
libraries, along with 1,707 human-written test cases for execution. Our NL-Code
pairs are harvested from StackOverflow forums to encourage natural and
practical coding queries. Moreover, ODEX supports four natural languages as
intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing
behavioral differences among top-performing code language models (LM). While
CODEX achieves better overall results, CODEGEN improves effectively via scaling
-- CODEGEN 6.1B performs comparably with CODEX 12B. Both models show
substantial gaps between open and closed domains, but CODEGEN gaps tend to
decrease with model size while CODEX gaps increase. We release ODEX to
facilitate research into open-domain problems for the code generation
community.
