  LLMs are being increasingly used for planning-style tasks, but their
capabilities for planning and reasoning are poorly understood. We present
AutoPlanBench, a novel method for automatically converting planning benchmarks
written in PDDL into textual descriptions and offer a benchmark dataset created
with our method. We show that while the best LLM planners do well on some
planning tasks, others remain out of reach of current methods.
