  The primary limitation of large language models (LLMs) is their restricted
understanding of the world. This poses significant difficulties for LLM-based
agents, particularly in domains where pre-trained LLMs lack sufficient
knowledge. In this paper, we introduce a novel framework, called AutoGuide,
that bridges the knowledge gap in pre-trained LLMs by leveraging implicit
knowledge in offline experiences. Specifically, AutoGuide effectively extracts
knowledge embedded in offline data by extracting a set of state-aware
guidelines. Importantly, each state-aware guideline is expressed in concise
natural language and follows a conditional structure, clearly describing the
state where it is applicable. As such, the resulting guidelines enable a
principled way to provide helpful knowledge pertinent to an agent's current
decision-making process. We show that our approach outperforms competitive
LLM-based baselines by a large margin in sequential decision-making benchmarks.
