  In this work we proposing adapting the Minecraft builder task into an LLM
benchmark suitable for evaluating LLM ability in spatially orientated tasks,
and informing builder agent design. Previous works have proposed corpora with
varying complex structures, and human written instructions. We instead attempt
to provide a comprehensive synthetic benchmark for testing builder agents over
a series of distinct tasks that comprise of common building operations. We
believe this approach allows us to probe specific strengths and weaknesses of
different agents, and test the ability of LLMs in the challenging area of
spatial reasoning and vector based math.
