  Large language models (LLMs) encode a vast amount of world knowledge acquired
from massive text datasets. Recent studies have demonstrated that LLMs can
assist an embodied agent in solving complex sequential decision making tasks by
providing high-level instructions. However, interactions with LLMs can be
time-consuming. In many practical scenarios, it requires a significant amount
of storage space that can only be deployed on remote cloud servers.
Additionally, using commercial LLMs can be costly since they may charge based
on usage frequency. In this paper, we explore how to enable intelligent
cost-effective interactions between a down stream task oriented agent and an
LLM. We find that this problem can be naturally formulated by a Markov decision
process (MDP), and propose When2Ask, a reinforcement learning based approach
that learns when it is necessary to query LLMs for high-level instructions to
accomplish a target task. On one side, When2Ask discourages unnecessary
redundant interactions, while on the other side, it enables the agent to
identify and follow useful instructions from the LLM. This enables the agent to
halt an ongoing plan and transition to a more suitable one based on new
environmental observations. Experiments on MiniGrid and Habitat environments
that entail planning sub-goals demonstrate that When2Ask learns to solve target
tasks with only a few necessary interactions with the LLM, significantly
reducing interaction costs in testing environments compared with baseline
methods. Our code is available at: https://github.com/ZJLAB-AMMI/LLM4RL.
