  Planning in a text-based environment continues to be a major challenge for AI
systems. Recent approaches have used language models to predict a planning
domain definition (e.g., PDDL) but have only been evaluated in closed-domain
simulated environments. To address this, we present Proc2PDDL , the first
dataset containing open-domain procedural texts paired with expert-annotated
PDDL representations. Using this dataset, we evaluate state-of-the-art models
on defining the preconditions and effects of actions. We show that Proc2PDDL is
highly challenging, with GPT-3.5's success rate close to 0% and GPT-4's around
35%. Our analysis shows both syntactic and semantic errors, indicating LMs'
deficiency in both generating domain-specific prgorams and reasoning about
events. We hope this analysis and dataset helps future progress towards
integrating the best of LMs and formal planning.
