  Embodied agents have achieved prominent performance in following human
instructions to complete tasks. However, the potential of providing
instructions informed by texts and images to assist humans in completing tasks
remains underexplored. To uncover this capability, we present the multimodal
procedural planning (MPP) task, in which models are given a high-level goal and
generate plans of paired text-image steps, providing more complementary and
informative guidance than unimodal plans. The key challenges of MPP are to
ensure the informativeness, temporal coherence,and accuracy of plans across
modalities. To tackle this, we propose Text-Image Prompting (TIP), a
dual-modality prompting method that jointly leverages zero-shot reasoning
ability in large language models (LLMs) and compelling text-to-image generation
ability from diffusion-based models. TIP improves the interaction in the dual
modalities using Text-to-Image Bridge and Image-to-Text Bridge, allowing LLMs
to guide the textual-grounded image plan generation and leveraging the
descriptions of image plans to ground the textual plan reversely. To address
the lack of relevant datasets, we collect WIKIPLAN and RECIPEPLAN as a testbed
for MPP. Our results show compelling human preferences and automatic scores
against unimodal and multimodal baselines on WIKIPLAN and RECIPEPLAN in terms
of informativeness, temporal coherence, and plan accuracy. Our code and data:
https://github.com/YujieLu10/MPP.
