  This paper focuses on extending the success of large language models (LLMs)
to sequential decision making. Existing efforts either (i) re-train or finetune
LLMs for decision making, or (ii) design prompts for pretrained LLMs. The
former approach suffers from the computational burden of gradient updates, and
the latter approach does not show promising results. In this paper, we propose
a new approach that leverages online model selection algorithms to efficiently
incorporate LLMs agents into sequential decision making. Statistically, our
approach significantly outperforms both traditional decision making algorithms
and vanilla LLM agents. Computationally, our approach avoids the need for
expensive gradient updates of LLMs, and throughout the decision making process,
it requires only a small number of LLM calls. We conduct extensive experiments
to verify the effectiveness of our proposed approach. As an example, on a
large-scale Amazon dataset, our approach achieves more than a $6$x performance
gain over baselines while calling LLMs in only $1.5$\% of the time steps.
