  Advancements in large language models (LLMs) have demonstrated their
potential in facilitating high-level reasoning, logical reasoning and robotics
planning. Recently, LLMs have also been able to generate reward functions for
low-level robot actions, effectively bridging the interface between high-level
planning and low-level robot control. However, the challenge remains that even
with syntactically correct plans, robots can still fail to achieve their
intended goals due to imperfect plans or unexpected environmental issues. To
overcome this, Vision Language Models (VLMs) have shown remarkable success in
tasks such as visual question answering. Leveraging the capabilities of VLMs,
we present a novel framework called Robotic Replanning with Perception and
Language Models (RePLan) that enables online replanning capabilities for
long-horizon tasks. This framework utilizes the physical grounding provided by
a VLM's understanding of the world's state to adapt robot actions when the
initial plan fails to achieve the desired goal. We developed a Reasoning and
Control (RC) benchmark with eight long-horizon tasks to test our approach. We
find that RePLan enables a robot to successfully adapt to unforeseen obstacles
while accomplishing open-ended, long-horizon goals, where baseline models
cannot, and can be readily applied to real robots. Find more information at
https://replan-lm.github.io/replan.github.io/
