  In image-based robot manipulation tasks with large observation and action
spaces, reinforcement learning struggles with low sample efficiency, slow
training speed, and uncertain convergence. As an alternative, large pre-trained
foundation models have shown promise in robotic manipulation, particularly in
zero-shot and few-shot applications. However, using these models directly is
unreliable due to limited reasoning capabilities and challenges in
understanding physical and spatial contexts. This paper introduces ExploRLLM, a
novel approach that leverages the inductive bias of foundation models (e.g.
Large Language Models) to guide exploration in reinforcement learning. We also
exploit these foundation models to reformulate the action and observation
spaces to enhance the training efficiency in reinforcement learning. Our
experiments demonstrate that guided exploration enables much quicker
convergence than training without it. Additionally, we validate that ExploRLLM
outperforms vanilla foundation model baselines and that the policy trained in
simulation can be applied in real-world settings without additional training.
