  Large language models (LLMs) encode a vast amount of semantic knowledge and
possess remarkable understanding and reasoning capabilities. Previous work has
explored how to ground LLMs in robotic tasks to generate feasible and
executable textual plans. However, low-level execution in the physical world
may deviate from the high-level textual plan due to environmental perturbations
or imperfect controller design. In this paper, we propose \textbf{DoReMi}, a
novel language model grounding framework that enables immediate Detection and
Recovery from Misalignments between plan and execution. Specifically, we
leverage LLMs to play a dual role, aiding not only in high-level planning but
also generating constraints that can indicate misalignment during execution.
Then vision language models (VLMs) are utilized to detect constraint violations
continuously. Our pipeline can monitor the low-level execution and enable
timely recovery if certain plan-execution misalignment occurs. Experiments on
various complex tasks including robot arms and humanoid robots demonstrate that
our method can lead to higher task success rates and shorter task completion
times. Videos of DoReMi are available at
\url{https://sites.google.com/view/doremi-paper}.
