  In this work we consider Code World Models, world models generated by a Large
Language Model (LLM) in the form of Python code for model-based Reinforcement
Learning (RL). Calling code instead of LLMs for planning has the advantages of
being precise, reliable, interpretable, and extremely efficient. However,
writing appropriate Code World Models requires the ability to understand
complex instructions, to generate exact code with non-trivial logic and to
self-debug a long program with feedback from unit tests and environment
trajectories. To address these challenges, we propose Generate, Improve and Fix
with Monte Carlo Tree Search (GIF-MCTS), a new code generation strategy for
LLMs. To test our approach, we introduce the Code World Models Benchmark
(CWMB), a suite of program synthesis and planning tasks comprised of 18 diverse
RL environments paired with corresponding textual descriptions and curated
trajectories. GIF-MCTS surpasses all baselines on the CWMB and two other
benchmarks, and we show that the Code World Models synthesized with it can be
successfully used for planning, resulting in model-based RL agents with greatly
improved sample efficiency and inference speed.
