  Humans write code in a fundamentally interactive manner and rely on constant
execution feedback to correct errors, resolve ambiguities, and decompose tasks.
While LLMs have recently exhibited promising coding capabilities, current
coding benchmarks mostly consider a static instruction-to-code sequence
transduction process, which has the potential for error propagation and a
disconnect between the generated code and its final execution environment. To
address this gap, we introduce InterCode, a lightweight, flexible, and
easy-to-use framework of interactive coding as a standard reinforcement
learning (RL) environment, with code as actions and execution feedback as
observations. Our framework is language and platform agnostic, uses
self-contained Docker environments to provide safe and reproducible execution,
and is compatible out-of-the-box with traditional seq2seq coding methods, while
enabling the development of new methods for interactive code generation. We use
InterCode to create three interactive code environments with Bash, SQL, and
Python as action spaces, leveraging data from the static NL2Bash, Spider, and
MBPP datasets. We demonstrate InterCode's viability as a testbed by evaluating
multiple state-of-the-art LLMs configured with different prompting strategies
such as ReAct and Plan & Solve. Our results showcase the benefits of
interactive code generation and demonstrate that InterCode can serve as a
challenging benchmark for advancing code understanding and generation
capabilities. InterCode is designed to be easily extensible and can even be
used to create new tasks such as Capture the Flag, a popular coding puzzle that
is inherently multi-step and involves multiple programming languages. Project
site with code and data: https://intercode-benchmark.github.io
