"My satire is more than just a joke, it's a call to action, and I've got the power\n",
            "I'm the one who's really making a difference, and you're just a fleeting flower.\n",
            "\n",
            "[The crowd continues to cheer and chant as the two comedians continue their rap battle.]\n"
          ]
        }
      ],
      "source": [
        "const response = await ollamaLlm.invoke(\"Simulate a rap battle between Stephen Colbert and John Oliver\");\n",
        "console.log(response.content);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "See the LangSmith trace [here](https://smith.langchain.com/public/31c178b5-4bea-4105-88c3-7ec95325c817/r)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Using in a chain\n",
        "\n",
        "We can create a summarization chain with either model by passing in the retrieved docs and a simple prompt.\n",
        "\n",
        "It formats the prompt template using the input key values provided and passes the formatted string to `LLama-V2`, or another specified LLM."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "import { RunnableSequence } from \"@langchain/core/runnables\";\n",
        "import { StringOutputParser } from \"@langchain/core/output_parsers\";\n",
        "import { PromptTemplate } from \"@langchain/core/prompts\";\n",
        "import { createStuffDocumentsChain } from \"langchain/chains/combine_documents\";\n",
        "\n",
        "const prompt = PromptTemplate.fromTemplate(\"Summarize the main themes in these retrieved docs: {context}\");\n",
        "\n",
        "const chain = await createStuffDocumentsChain({\n",
        "  llm: ollamaLlm,\n",
        "  outputParser: new StringOutputParser(),\n",
        "  prompt,\n",
        "})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u001b[32m\"The main themes retrieved from the provided documents are:\\n\"\u001b[39m +\n",
              "  \u001b[32m\"\\n\"\u001b[39m +\n",
              "  \u001b[32m\"1. Sensory Memory: The ability to retain\"\u001b[39m... 1117 more characters"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "const question = \"What are the approaches to Task Decomposition?\";\n",
        "const docs = await vectorStore.similaritySearch(question);\n",
        "await chain.invoke({\n",
        "  context: docs,\n",
        "});"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "See the LangSmith trace [here](https://smith.langchain.com/public/47cf6c2a-3d86-4f2b-9a51-ee4663b19152/r)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Q&A \n",
        "\n",
        "We can also use the LangChain Prompt Hub to store and fetch prompts that are model-specific.\n",
        "\n",
        "Let's try with a default RAG prompt, [here](https://smith.langchain.com/hub/rlm/rag-prompt)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "import { pull } from \"langchain/hub\";\n",
        "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
        "\n",
        "const ragPrompt = await pull<ChatPromptTemplate>(\"rlm/rag-prompt\");\n",
        "\n",
        "const chain = await createStuffDocumentsChain({\n",
        "  llm: ollamaLlm,\n",
        "  outputParser: new StringOutputParser(),\n",
        "  prompt: ragPrompt,\n",
        "});"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see what this prompt actually looks like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\n",
            "Question: {question} \n",
            "Context: {context} \n",
            "Answer:\n"
          ]
        }
      ],
      "source": [
        "console.log(ragPrompt.promptMessages.map((msg) => msg.prompt.template).join(\"\\n\"));"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u001b[32m\"Task decomposition is a crucial step in breaking down complex problems into manageable parts for eff\"\u001b[39m... 1095 more characters"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "await chain.invoke({ context: docs, question });"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "See the LangSmith trace [here](https://smith.langchain.com/public/dd3a189b-53a1-4f31-9766-244cd04ad1f7/r)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Q&A with retrieval\n",
        "\n",
        "Instead of manually passing in docs, we can automatically retrieve them from our vector store based on the user question.\n",
        "\n",
        "This will use a QA default prompt and will retrieve from the vectorDB."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "\u001b[32m\"Based on the context provided, I understand that you are asking me to answer a question related to m\"\u001b[39m... 948 more characters"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import { RunnablePassthrough, RunnableSequence } from \"@langchain/core/runnables\";\n",
        "import { formatDocumentsAsString } from \"langchain/util/document\";\n",
        "\n",
        "const retriever = vectorStore.asRetriever();\n",
        "\n",
        "const qaChain = RunnableSequence.from([\n",
        "  {\n",
        "    context: (input: { question: string }, callbacks) => {\n",
        "      const retrieverAndFormatter = retriever.pipe(formatDocumentsAsString);\n",
        "      return retrieverAndFormatter.invoke(input.question, callbacks);\n",
        "    },\n",
        "    question: new RunnablePassthrough(),\n",
        "  },\n",
        "  ragPrompt,\n",
        "  ollamaLlm,\n",
        "  new StringOutputParser(),\n",
        "]);\n",
        "\n",
        "await qaChain.invoke({ question });"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "See the LangSmith trace [here](https://smith.langchain.com/public/440e65ee-0301-42cf-afc9-f09cfb52cf64/r)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Deno",
      "language": "typescript",
      "name": "deno"
    },
    "language_info": {
      "file_extension": ".ts",
      "mimetype": "text/x.typescript",
      "name": "typescript",
      "nb_converter": "script",


"  [\"human\", \"{input}\"],\n",
    "]);\n",
    "\n",
    "const questionAnswerChain = await createStuffDocumentsChain({\n",
    "  llm,\n",
    "  prompt,\n",
    "});\n",
    "\n",
    "const ragChain = await createRetrievalChain({\n",
    "  retriever,\n",
    "  combineDocsChain: questionAnswerChain,\n",
    "});"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bf55faaf-0d17-4b74-925d-c478b555f7b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task decomposition involves breaking down large and complex tasks into smaller, more manageable subgoals or steps. This approach helps agents or models efficiently handle intricate tasks by simplifying them into easier components. Task decomposition can be achieved through techniques like Chain of Thought, Tree of Thoughts, or by using task-specific instructions and human input.\n"
     ]
    }
   ],
   "source": [
    "const response = await ragChain.invoke({ input: \"What is Task Decomposition?\" });\n",
    "console.log(response.answer);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "187404c7-db47-49c5-be29-9ecb96dc9afa",
   "metadata": {},
   "source": [
    "Note that we have used the built-in chain constructors `createStuffDocumentsChain` and `createRetrievalChain`, so that the basic ingredients to our solution are:\n",
    "\n",
    "1. retriever;\n",
    "2. prompt;\n",
    "3. LLM.\n",
    "\n",
    "This will simplify the process of incorporating chat history.\n",
    "\n",
    "### Adding chat history\n",
    "\n",
    "The chain we have built uses the input query directly to retrieve relevant context. But in a conversational setting, the user query might require conversational context to be understood. For example, consider this exchange:\n",
    "\n",
    "> Human: \"What is Task Decomposition?\"\n",
    ">\n",
    "> AI: \"Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model.\"\n",
    ">\n",
    "> Human: \"What are common ways of doing it?\"\n",
    "\n",
    "In order to answer the second question, our system needs to understand that \"it\" refers to \"Task Decomposition.\"\n",
    "\n",
    "We'll need to update two things about our existing app:\n",
    "\n",
    "1. **Prompt**: Update our prompt to support historical messages as an input.\n",
    "2. **Contextualizing questions**: Add a sub-chain that takes the latest user question and reformulates it in the context of the chat history. This can be thought of simply as building a new \"history aware\" retriever. Whereas before we had:\n",
    "   - `query` -> `retriever`  \n",
    "     Now we will have:\n",
    "   - `(query, conversation history)` -> `LLM` -> `rephrased query` -> `retriever`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776ae958-cbdc-4471-8669-c6087436f0b5",
   "metadata": {},
   "source": [
    "#### Contextualizing the question\n",
    "\n",
    "First we'll need to define a sub-chain that takes historical messages and the latest user question, and reformulates the question if it makes reference to any information in the historical information.\n",
    "\n",
    "We'll use a prompt that includes a `MessagesPlaceholder` variable under the name \"chat_history\". This allows us to pass in a list of Messages to the prompt using the \"chat_history\" input key, and these messages will be inserted after the system message and before the human message containing the latest question.\n",
    "\n",
    "Note that we leverage a helper function [createHistoryAwareRetriever](https://api.js.langchain.com/functions/langchain.chains_history_aware_retriever.createHistoryAwareRetriever.html) for this step, which manages the case where `chat_history` is empty, and otherwise applies `prompt.pipe(llm).pipe(new StringOutputParser()).pipe(retriever)` in sequence.\n",
    "\n",
    "`createHistoryAwareRetriever` constructs a chain that accepts keys `input` and `chat_history` as input, and has the same output schema as a retriever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2b685428-8b82-4af1-be4f-7232c5d55b73",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { createHistoryAwareRetriever } from \"langchain/chains/history_aware_retriever\";\n",
    "import { MessagesPlaceholder } from \"@langchain/core/prompts\";\n",
    "\n",
    "const contextualizeQSystemPrompt = \n",
    "  \"Given a chat history and the latest user question \" +\n",
    "  \"which might reference context in the chat history, \" +\n",
    "  \"formulate a standalone question which can be understood \" +\n",
    "  \"without the chat history. Do NOT answer the question, \" +\n",
    "  \"just reformulate it if needed and otherwise return it as is.\";\n",
    "\n",
    "const contextualizeQPrompt = ChatPromptTemplate.fromMessages([\n",
    "  [\"system\", contextualizeQSystemPrompt],\n",
    "  new MessagesPlaceholder(\"chat_history\"),\n",
    "  [\"human\", \"{input}\"],\n",
    "]);\n",
    "\n",
    "const historyAwareRetriever = await createHistoryAwareRetriever({\n",
    "  llm,\n",
    "  retriever,\n",
    "  rephrasePrompt: contextualizeQPrompt,\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a47168-4a1f-4e39-bd2d-d5b03609a243",
   "metadata": {},
   "source": [
    "This chain prepends a rephrasing of the input query to our retriever, so that the retrieval incorporates the context of the conversation.\n",
    "\n",
    "Now we can build our full QA chain. This is as simple as updating the retriever to be our new `historyAwareRetriever`.\n",
    "\n",
    "Again, we will use [createStuffDocumentsChain](https://api.js.langchain.com/functions/langchain.chains_combine_documents.createStuffDocumentsChain.html) to generate a `questionAnswerChain2`, with input keys `context`, `chat_history`, and `input`-- it accepts the retrieved context alongside the conversation history and query to generate an answer. A more detailed explaination is over [here](/docs/tutorials/rag/#built-in-chains)\n",
    "\n",
    "We build our final `ragChain2` with [createRetrievalChain](https://api.js.langchain.com/functions/langchain.chains_retrieval.createRetrievalChain.html). This chain applies the `historyAwareRetriever` and `questionAnswerChain2` in sequence, retaining intermediate outputs such as the retrieved context for convenience. It has input keys `input` and `chat_history`, and includes `input`, `chat_history`, `context`, and `answer` in its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "66f275f3-ddef-4678-b90d-ee64576878f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "const qaPrompt = ChatPromptTemplate.fromMessages([\n",
    "  [\"system\", systemPrompt],\n",
    "  new MessagesPlaceholder(\"chat_history\"),\n",
    "  [\"human\", \"{input}\"],\n",
    "]);\n",
    "\n",
    "const questionAnswerChain2 = await createStuffDocumentsChain({\n",
    "  llm,\n",
    "  prompt: qaPrompt,\n",
    "});\n",
    "\n",
    "const ragChain2 = await createRetrievalChain({\n",
    "  retriever: historyAwareRetriever,\n",
    "  combineDocsChain: questionAnswerChain2,\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba1ae56-7ecb-4563-b792-50a1a5042df3",
   "metadata": {},
   "source": [


"{ answer: ' using' }\n",
      "----\n",
      "{ answer: ' task' }\n",
      "----\n",
      "{ answer: '-specific' }\n",
      "----\n",
      "{ answer: ' instructions' }\n",
      "----\n",
      "{ answer: '.' }\n",
      "----\n",
      "{ answer: '' }\n",
      "----\n",
      "{ answer: '' }\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "import { CheerioWebBaseLoader } from \"@langchain/community/document_loaders/web/cheerio\";\n",
    "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
    "import { MemoryVectorStore } from \"langchain/vectorstores/memory\";\n",
    "import { OpenAIEmbeddings, ChatOpenAI } from \"@langchain/openai\";\n",
    "import { ChatPromptTemplate, MessagesPlaceholder } from \"@langchain/core/prompts\";\n",
    "import { createHistoryAwareRetriever } from \"langchain/chains/history_aware_retriever\";\n",
    "import { createStuffDocumentsChain } from \"langchain/chains/combine_documents\";\n",
    "import { createRetrievalChain } from \"langchain/chains/retrieval\";\n",
    "import { RunnableWithMessageHistory } from \"@langchain/core/runnables\";\n",
    "import { ChatMessageHistory } from \"langchain/stores/message/in_memory\";\n",
    "import { BaseChatMessageHistory } from \"@langchain/core/chat_history\";\n",
    "\n",
    "const llm2 = new ChatOpenAI({ model: \"gpt-3.5-turbo\", temperature: 0 });\n",
    "\n",
    "// Construct retriever\n",
    "const loader2 = new CheerioWebBaseLoader(\n",
    "  \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
    "  {\n",
    "    selector: \".post-content, .post-title, .post-header\"\n",
    "  }\n",
    ");\n",
    "\n",
    "const docs2 = await loader2.load();\n",
    "\n",
    "const textSplitter2 = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 });\n",
    "const splits2 = await textSplitter2.splitDocuments(docs2);\n",
    "const vectorstore2 = await MemoryVectorStore.fromDocuments(splits2, new OpenAIEmbeddings());\n",
    "const retriever2 = vectorstore2.asRetriever();\n",
    "\n",
    "// Contextualize question\n",
    "const contextualizeQSystemPrompt2 = \n",
    "  \"Given a chat history and the latest user question \" +\n",
    "  \"which might reference context in the chat history, \" +\n",
    "  \"formulate a standalone question which can be understood \" +\n",
    "  \"without the chat history. Do NOT answer the question, \" +\n",
    "  \"just reformulate it if needed and otherwise return it as is.\";\n",
    "\n",
    "const contextualizeQPrompt2 = ChatPromptTemplate.fromMessages([\n",
    "  [\"system\", contextualizeQSystemPrompt2],\n",
    "  new MessagesPlaceholder(\"chat_history\"),\n",
    "  [\"human\", \"{input}\"],\n",
    "]);\n",
    "\n",
    "const historyAwareRetriever2 = await createHistoryAwareRetriever({\n",
    "  llm: llm2,\n",
    "  retriever: retriever2,\n",
    "  rephrasePrompt: contextualizeQPrompt2\n",
    "});\n",
    "\n",
    "// Answer question\n",
    "const systemPrompt2 = \n",
    "  \"You are an assistant for question-answering tasks. \" +\n",
    "  \"Use the following pieces of retrieved context to answer \" +\n",
    "  \"the question. If you don't know the answer, say that you \" +\n",
    "  \"don't know. Use three sentences maximum and keep the \" +\n",
    "  \"answer concise.\" +\n",
    "  \"\\n\\n\" +\n",
    "  \"{context}\";\n",
    "\n",
    "const qaPrompt2 = ChatPromptTemplate.fromMessages([\n",
    "  [\"system\", systemPrompt2],\n",
    "  new MessagesPlaceholder(\"chat_history\"),\n",
    "  [\"human\", \"{input}\"],\n",
    "]);\n",
    "\n",
    "const questionAnswerChain3 = await createStuffDocumentsChain({\n",
    "  llm,\n",
    "  prompt: qaPrompt2,\n",
    "});\n",
    "\n",
    "const ragChain3 = await createRetrievalChain({\n",
    "  retriever: historyAwareRetriever2,\n",
    "  combineDocsChain: questionAnswerChain3,\n",
    "});\n",
    "\n",
    "// Statefully manage chat history\n",
    "const store2: Record<string, BaseChatMessageHistory> = {};\n",
    "\n",
    "function getSessionHistory2(sessionId: string): BaseChatMessageHistory {\n",
    "  if (!(sessionId in store2)) {\n",
    "    store2[sessionId] = new ChatMessageHistory();\n",
    "  }\n",
    "  return store2[sessionId];\n",
    "}\n",
    "\n",
    "const conversationalRagChain2 = new RunnableWithMessageHistory({\n",
    "  runnable: ragChain3,\n",
    "  getMessageHistory: getSessionHistory2,\n",
    "  inputMessagesKey: \"input\",\n",
    "  historyMessagesKey: \"chat_history\",\n",
    "  outputMessagesKey: \"answer\",\n",
    "});\n",
    "\n",
    "// Example usage\n",
    "const query2 = \"What is Task Decomposition?\";\n",
    "\n",
    "for await (const s of await conversationalRagChain2.stream(\n",
    "  { input: query2 },\n",
    "  { configurable: { sessionId: \"unique_session_id\" } }\n",
    ")) {\n",
    "  console.log(s);\n",
    "  console.log(\"----\");\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861da8ed-d890-4fdc-a3bf-30433db61e0d",
   "metadata": {},
   "source": [
    "## Agents {#agents}\n",
    "\n",
    "Agents leverage the reasoning capabilities of LLMs to make decisions during execution. Using agents allow you to offload some discretion over the retrieval process. Although their behavior is less predictable than chains, they offer some advantages in this context:\n",
    "\n",
    "- Agents generate the input to the retriever directly, without necessarily needing us to explicitly build in contextualization, as we did above;\n",
    "- Agents can execute multiple retrieval steps in service of a query, or refrain from executing a retrieval step altogether (e.g., in response to a generic greeting from a user).\n",
    "\n",
    "### Retrieval tool\n",
    "\n",
    "Agents can access \"tools\" and manage their execution. In this case, we will convert our retriever into a LangChain tool to be wielded by the agent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "809cc747-2135-40a2-8e73-e4556343ee64",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { createRetrieverTool } from \"langchain/tools/retriever\";\n",
    "\n",
    "const tool = createRetrieverTool(\n",
    "    retriever,\n",
    "    {\n",
    "      name: \"blog_post_retriever\",\n",
    "      description: \"Searches and returns excerpts from the Autonomous Agents blog post.\",\n",
    "    }\n",
    ")\n",
    "const tools = [tool]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07dcb968-ed9a-458a-85e1-528cd28c6965",
   "metadata": {},
   "source": [
    "Tools are LangChain [Runnables](/docs/concepts#langchain-expression-language-lcel), and implement the usual interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "931c4fe3-c603-4efb-9b37-5f7cbbb1cbbd",
   "metadata": {},


{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "sidebar_position: 1\n",
    "keywords: [conversationchain]\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a Chatbot\n",
    "\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [Chat Models](/docs/concepts/#chat-models)\n",
    "- [Prompt Templates](/docs/concepts/#prompt-templates)\n",
    "- [Chat History](/docs/concepts/#chat-history)\n",
    "\n",
    "This guide requires `langgraph >= 0.2.28`.\n",
    "\n",
    ":::\n",
    "\n",
    "\n",
    "```{=mdx}\n",
    "\n",
    ":::note\n",
    "\n",
    "This tutorial previously built a chatbot using [RunnableWithMessageHistory](https://api.js.langchain.com/classes/_langchain_core.runnables.RunnableWithMessageHistory.html). You can access this version of the tutorial in the [v0.2 docs](https://js.langchain.com/v0.2/docs/tutorials/chatbot/).\n",
    "\n",
    "The LangGraph implementation offers a number of advantages over `RunnableWithMessageHistory`, including the ability to persist arbitrary components of an application's state (instead of only messages).\n",
    "\n",
    ":::\n",
    "\n",
    "```\n",
    "\n",
    "## Overview\n",
    "\n",
    "We'll go over an example of how to design and implement an LLM-powered chatbot. \n",
    "This chatbot will be able to have a conversation and remember previous interactions.\n",
    "\n",
    "\n",
    "Note that this chatbot that we build will only use the language model to have a conversation.\n",
    "There are several other related concepts that you may be looking for:\n",
    "\n",
    "- [Conversational RAG](/docs/tutorials/qa_chat_history): Enable a chatbot experience over an external source of data\n",
    "- [Agents](https://langchain-ai.github.io/langgraphjs/tutorials/multi_agent/agent_supervisor/): Build a chatbot that can take actions\n",
    "\n",
    "This tutorial will cover the basics which will be helpful for those two more advanced topics, but feel free to skip directly to there should you choose.\n",
    "\n",
    "## Setup\n",
    "\n",
    "### Jupyter Notebook\n",
    "\n",
    "This guide (and most of the other guides in the documentation) uses [Jupyter notebooks](https://jupyter.org/) and assumes the reader is as well. Jupyter notebooks are perfect for learning how to work with LLM systems because oftentimes things can go wrong (unexpected output, API down, etc) and going through guides in an interactive environment is a great way to better understand them.\n",
    "\n",
    "This and other tutorials are perhaps most conveniently run in a Jupyter notebook. See [here](https://jupyter.org/install) for instructions on how to install.\n",
    "\n",
    "### Installation\n",
    "\n",
    "For this tutorial we will need `@langchain/core` and `langgraph`:\n",
    "\n",
    "```{=mdx}\n",
    "import Npm2Yarn from \"@theme/Npm2Yarn\"\n",
    "\n",
    "<Npm2Yarn>\n",
    "  @langchain/core @langchain/langgraph uuid\n",
    "</Npm2Yarn>\n",
    "```\n",
    "\n",
    "For more details, see our [Installation guide](/docs/how_to/installation).\n",
    "\n",
    "### LangSmith\n",
    "\n",
    "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.\n",
    "As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.\n",
    "The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
    "\n",
    "After you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
    "\n",
    "```typescript\n",
    "process.env.LANGCHAIN_TRACING_V2 = \"true\"\n",
    "process.env.LANGCHAIN_API_KEY = \"...\"\n",
    "```\n",
    "\n",
    "## Quickstart\n",
    "\n",
    "First up, let's learn how to use a language model by itself. LangChain supports many different language models that you can use interchangeably - select the one you want to use below!\n",
    "\n",
    "```{=mdx}\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs customVarName=\"llm\" />\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "// @lc-docs-hide-cell\n",
    "\n",
    "import { ChatOpenAI } from \"@langchain/openai\";\n",
    "\n",
    "const llm = new ChatOpenAI({ model: \"gpt-4o-mini\" })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first use the model directly. `ChatModel`s are instances of LangChain \"Runnables\", which means they expose a standard interface for interacting with them. To just simply call the model, we can pass in a list of messages to the `.invoke` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIMessage {\n",
      "  \"id\": \"chatcmpl-ABUXeSO4JQpxO96lj7iudUptJ6nfW\",\n",
      "  \"content\": \"Hi Bob! How can I assist you today?\",\n",
      "  \"additional_kwargs\": {},\n",
      "  \"response_metadata\": {\n",
      "    \"tokenUsage\": {\n",
      "      \"completionTokens\": 10,\n",
      "      \"promptTokens\": 10,\n",
      "      \"totalTokens\": 20\n",
      "    },\n",
      "    \"finish_reason\": \"stop\",\n",
      "    \"system_fingerprint\": \"fp_1bb46167f9\"\n",
      "  },\n",
      "  \"tool_calls\": [],\n",
      "  \"invalid_tool_calls\": [],\n",
      "  \"usage_metadata\": {\n",
      "    \"input_tokens\": 10,\n",
      "    \"output_tokens\": 10,\n",
      "    \"total_tokens\": 20\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "await llm.invoke([{ role: \"user\", content: \"Hi im bob\" }])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model on its own does not have any concept of state. For example, if you ask a followup question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIMessage {\n",
      "  \"id\": \"chatcmpl-ABUXe1Zih4gMe3XgotWL83xeWub2h\",\n",
      "  \"content\": \"I'm sorry, but I don't have access to personal information about individuals unless it has been shared with me during our conversation. If you'd like to tell me your name, feel free to do so!\",\n",
      "  \"additional_kwargs\": {},\n",
      "  \"response_metadata\": {\n",
      "    \"tokenUsage\": {\n",
      "      \"completionTokens\": 39,\n",
      "      \"promptTokens\": 10,\n",
      "      \"totalTokens\": 49\n",
      "    },\n",
      "    \"finish_reason\": \"stop\",\n",
      "    \"system_fingerprint\": \"fp_1bb46167f9\"\n",
      "  },\n",
      "  \"tool_calls\": [],\n",
      "  \"invalid_tool_calls\": [],\n",


"  \"usage_metadata\": {\n",
      "    \"input_tokens\": 10,\n",
      "    \"output_tokens\": 39,\n",
      "    \"total_tokens\": 49\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "await llm.invoke([{ role: \"user\", content: \"Whats my name\" }])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the example [LangSmith trace](https://smith.langchain.com/public/3b768e44-a319-453a-bd6e-30f9df75f16a/r)\n",
    "\n",
    "We can see that it doesn't take the previous conversation turn into context, and cannot answer the question.\n",
    "This makes for a terrible chatbot experience!\n",
    "\n",
    "To get around this, we need to pass the entire conversation history into the model. Let's see what happens when we do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIMessage {\n",
      "  \"id\": \"chatcmpl-ABUXfX4Fnp247rOxyPlBUYMQgahj2\",\n",
      "  \"content\": \"Your name is Bob! How can I help you today?\",\n",
      "  \"additional_kwargs\": {},\n",
      "  \"response_metadata\": {\n",
      "    \"tokenUsage\": {\n",
      "      \"completionTokens\": 12,\n",
      "      \"promptTokens\": 33,\n",
      "      \"totalTokens\": 45\n",
      "    },\n",
      "    \"finish_reason\": \"stop\",\n",
      "    \"system_fingerprint\": \"fp_1bb46167f9\"\n",
      "  },\n",
      "  \"tool_calls\": [],\n",
      "  \"invalid_tool_calls\": [],\n",
      "  \"usage_metadata\": {\n",
      "    \"input_tokens\": 33,\n",
      "    \"output_tokens\": 12,\n",
      "    \"total_tokens\": 45\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "await llm.invoke([\n",
    "  { role: \"user\", content: \"Hi! I'm Bob\" },\n",
    "  { role: \"assistant\", content: \"Hello Bob! How can I assist you today?\" },\n",
    "  { role: \"user\", content: \"What's my name?\" }\n",
    "]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can see that we get a good response!\n",
    "\n",
    "This is the basic idea underpinning a chatbot's ability to interact conversationally.\n",
    "So how do we best implement this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Message persistence\n",
    "\n",
    "[LangGraph](https://langchain-ai.github.io/langgraphjs/) implements a built-in persistence layer, making it ideal for chat applications that support multiple conversational turns.\n",
    "\n",
    "Wrapping our chat model in a minimal LangGraph application allows us to automatically persist the message history, simplifying the development of multi-turn applications.\n",
    "\n",
    "LangGraph comes with a simple in-memory checkpointer, which we use below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import { START, END, MessagesAnnotation, StateGraph, MemorySaver } from \"@langchain/langgraph\";\n",
    "\n",
    "// Define the function that calls the model\n",
    "const callModel = async (state: typeof MessagesAnnotation.State) => {\n",
    "  const response = await llm.invoke(state.messages);\n",
    "  return { messages: response };\n",
    "};\n",
    "\n",
    "// Define a new graph\n",
    "const workflow = new StateGraph(MessagesAnnotation)\n",
    "  // Define the node and edge\n",
    "  .addNode(\"model\", callModel)\n",
    "  .addEdge(START, \"model\")\n",
    "  .addEdge(\"model\", END);\n",
    "\n",
    "// Add memory\n",
    "const memory = new MemorySaver();\n",
    "const app = workflow.compile({ checkpointer: memory });"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to create a `config` that we pass into the runnable every time. This config contains information that is not part of the input directly, but is still useful. In this case, we want to include a `thread_id`. This should look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import { v4 as uuidv4 } from \"uuid\";\n",
    "\n",
    "const config = { configurable: { thread_id: uuidv4() } };"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This enables us to support multiple conversation threads with a single application, a common requirement when your application has multiple users.\n",
    "\n",
    "We can then invoke the application:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIMessage {\n",
      "  \"id\": \"chatcmpl-ABUXfjqCno78CGXCHoAgamqXG1pnZ\",\n",
      "  \"content\": \"Hi Bob! How can I assist you today?\",\n",
      "  \"additional_kwargs\": {},\n",
      "  \"response_metadata\": {\n",
      "    \"tokenUsage\": {\n",
      "      \"completionTokens\": 10,\n",
      "      \"promptTokens\": 12,\n",
      "      \"totalTokens\": 22\n",
      "    },\n",
      "    \"finish_reason\": \"stop\",\n",
      "    \"system_fingerprint\": \"fp_1bb46167f9\"\n",
      "  },\n",
      "  \"tool_calls\": [],\n",
      "  \"invalid_tool_calls\": [],\n",
      "  \"usage_metadata\": {\n",
      "    \"input_tokens\": 12,\n",
      "    \"output_tokens\": 10,\n",
      "    \"total_tokens\": 22\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "const input = [\n",
    "  {\n",
    "    role: \"user\",\n",
    "    content: \"Hi! I'm Bob.\",\n",
    "  }\n",
    "]\n",
    "const output = await app.invoke({ messages: input }, config)\n",
    "// The output contains all messages in the state.\n",
    "// This will long the last message in the conversation.\n",
    "console.log(output.messages[output.messages.length - 1]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIMessage {\n",
      "  \"id\": \"chatcmpl-ABUXgzHFHk4KsaNmDJyvflHq4JY2L\",\n",
      "  \"content\": \"Your name is Bob! How can I help you today, Bob?\",\n",
      "  \"additional_kwargs\": {},\n",
      "  \"response_metadata\": {\n",
      "    \"tokenUsage\": {\n",
      "      \"completionTokens\": 14,\n",
      "      \"promptTokens\": 34,\n",
      "      \"totalTokens\": 48\n",
      "    },\n",
      "    \"finish_reason\": \"stop\",\n",
      "    \"system_fingerprint\": \"fp_1bb46167f9\"\n",
      "  },\n",
      "  \"tool_calls\": [],\n",
      "  \"invalid_tool_calls\": [],\n",
      "  \"usage_metadata\": {\n",
      "    \"input_tokens\": 34,\n",
      "    \"output_tokens\": 14,\n",


{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Build a Retrieval Augmented Generation (RAG) App\n",
        "\n",
        "One of the most powerful applications enabled by LLMs is sophisticated question-answering (Q&A) chatbots. These are applications that can answer questions about specific source information. These applications use a technique known as Retrieval Augmented Generation, or RAG.\n",
        "\n",
        "This tutorial will show how to build a simple Q&A application\n",
        "over a text data source. Along the way we’ll go over a typical Q&A\n",
        "architecture and highlight additional resources for more advanced Q&A techniques. We’ll also see\n",
        "how LangSmith can help us trace and understand our application.\n",
        "LangSmith will become increasingly helpful as our application grows in\n",
        "complexity.\n",
        "\n",
        "If you're already familiar with basic retrieval, you might also be interested in\n",
        "this [high-level overview of different retrieval techinques](/docs/concepts/#retrieval).\n",
        "\n",
        "## What is RAG?\n",
        "\n",
        "RAG is a technique for augmenting LLM knowledge with additional data.\n",
        "\n",
        "LLMs can reason about wide-ranging topics, but their knowledge is limited to the public data up to a specific point in time that they were trained on. If you want to build AI applications that can reason about private data or data introduced after a model's cutoff date, you need to augment the knowledge of the model with the specific information it needs. The process of bringing the appropriate information and inserting it into the model prompt is known as Retrieval Augmented Generation (RAG).\n",
        "\n",
        "LangChain has a number of components designed to help build Q&A applications, and RAG applications more generally. \n",
        "\n",
        "**Note**: Here we focus on Q&A for unstructured data. If you are interested for RAG over structured data, check out our tutorial on doing [question/answering over SQL data](/docs/tutorials/sql_qa).\n",
        "\n",
        "## Concepts\n",
        "A typical RAG application has two main components:\n",
        "\n",
        "**Indexing**: a pipeline for ingesting data from a source and indexing it. *This usually happens offline.*\n",
        "\n",
        "**Retrieval and generation**: the actual RAG chain, which takes the user query at run time and retrieves the relevant data from the index, then passes that to the model.\n",
        "\n",
        "The most common full sequence from raw data to answer looks like:\n",
        "\n",
        "### Indexing\n",
        "1. **Load**: First we need to load our data. This is done with [Document Loaders](/docs/concepts/#document-loaders).\n",
        "2. **Split**: [Text splitters](/docs/concepts/#text-splitters) break large `Documents` into smaller chunks. This is useful both for indexing data and for passing it in to a model, since large chunks are harder to search over and won't fit in a model's finite context window.\n",
        "3. **Store**: We need somewhere to store and index our splits, so that they can later be searched over. This is often done using a [VectorStore](/docs/concepts/#vectorstores) and [Embeddings](/docs/concepts/#embedding-models) model.\n",
        "\n",
        "![index_diagram](../../static/img/rag_indexing.png)\n",
        "\n",
        "### Retrieval and generation\n",
        "4. **Retrieve**: Given a user input, relevant splits are retrieved from storage using a [Retriever](/docs/concepts/#retrievers).\n",
        "5. **Generate**: A [ChatModel](/docs/concepts/#chat-models) / [LLM](/docs/concepts/#llms) produces an answer using a prompt that includes the question and the retrieved data\n",
        "\n",
        "![retrieval_diagram](../../static/img/rag_retrieval_generation.png)\n",
        "\n",
        "\n",
        "## Setup\n",
        "\n",
        "### Installation\n",
        "\n",
        "To install LangChain run:\n",
        "\n",
        "```bash npm2yarn\n",
        "npm i langchain @langchain/core\n",
        "```\n",
        "\n",
        "For more details, see our [Installation guide](/docs/how_to/installation).\n",
        "\n",
        "### LangSmith\n",
        "\n",
        "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.\n",
        "As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.\n",
        "The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
        "\n",
        "After you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
        "\n",
        "```shell\n",
        "export LANGCHAIN_TRACING_V2=\"true\"\n",
        "export LANGCHAIN_API_KEY=\"...\"\n",
        "\n",
        "# Reduce tracing latency if you are not in a serverless environment\n",
        "# export LANGCHAIN_CALLBACKS_BACKGROUND=true\n",
        "```\n",
        "\n",
        "```{=mdx}\n",
        "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
        "\n",
        "<ChatModelTabs customVarName=\"llm\" />\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Preview\n",
        "\n",
        "In this guide we’ll build a QA app over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng, which allows us to ask questions about the contents of the post.\n",
        "\n",
        "We can create a simple indexing pipeline and RAG chain to do this in only a few lines of code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "import \"cheerio\";\n",
        "import { CheerioWebBaseLoader } from \"@langchain/community/document_loaders/web/cheerio\";\n",
        "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
        "import { MemoryVectorStore } from \"langchain/vectorstores/memory\"\n",
        "import { OpenAIEmbeddings, ChatOpenAI } from \"@langchain/openai\";\n",
        "import { pull } from \"langchain/hub\";\n",
        "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
        "import { StringOutputParser } from \"@langchain/core/output_parsers\";\n",
        "import { createStuffDocumentsChain } from \"langchain/chains/combine_documents\";\n",
        "\n",
        "const loader = new CheerioWebBaseLoader(\n",
        "  \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
        ");\n",
        "\n",
        "const docs = await loader.load();\n",
        "\n",
        "const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 });\n",
        "const splits = await textSplitter.splitDocuments(docs);\n",
        "const vectorStore = await MemoryVectorStore.fromDocuments(splits, new OpenAIEmbeddings());\n",
        "\n",
        "// Retrieve and generate using the relevant snippets of the blog.\n",
        "const retriever = vectorStore.asRetriever();\n",
        "const prompt = await pull<ChatPromptTemplate>(\"rlm/rag-prompt\");\n",
        "const llm = new ChatOpenAI({ model: \"gpt-3.5-turbo\", temperature: 0 });\n",
        "\n",
        "const ragChain = await createStuffDocumentsChain({\n",
        "  llm,\n",
        "  prompt,\n",
        "  outputParser: new StringOutputParser(),\n",
        "})\n",
        "\n",
        "const retrievedDocs = await retriever.invoke(\"what is task decomposition\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The prompt looks like this:\n",
        "\n",
        "```\n",
        "You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\n",
        "Question: {question} \n",


"Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.Another quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.Self-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.ReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.The ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:In both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.Reflexion (Shinn & Labash 2023) is a framework to equips agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.The heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.Self-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.Chain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\{(x, y_i , r_i , z_i)\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\geq r_{n-1} \\geq \\dots \\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\tau_h = (x, z_i, y_i, z_j, y_j, \\dots, z_n, y_n)$, where $\\leq i \\leq j \\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.To avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.The training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.The idea of CoH is to present a history of sequentially improved outputs  in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.The paper hypothesizes that any algorithm that generates a set of learning histories can be distilled into a neural network by performing behavioral cloning over actions. The history data is generated by a set of source policies, each trained for a specific task. At the training stage, during each RL run, a random task is sampled and a subsequence of multi-episode history is used for training, such that the learned policy is task-agnostic.In reality, the model has limited context window length, so episodes should be short enough to construct multi-episode history. Multi-episodic contexts of 2-4 episodes are necessary to learn a near-optimal in-context RL algorithm. The emergence of in-context RL requires long enough context.In comparison with three baselines, including ED (expert distillation, behavior cloning with expert trajectories instead of learning history), source policy (used for generating trajectories for distillation by UCB), RL^2 (Duan et al. 2017; used as upper bound since it needs online RL), AD demonstrates in-context RL with performance getting close to RL^2 despite only using offline RL and learns much faster than other baselines. When conditioned on partial training history of the source policy, AD also improves much faster than ED baseline.(Big thank you to ChatGPT for helping me draft this section. I’ve learned a lot about the human brain and data structure for fast MIPS in my conversations with ChatGPT.)Memory can be defined as the processes used to acquire, store, retain, and later retrieve information. There are several types of memory in human brains.Sensory Memory: This is the earliest stage of memory, providing the ability to retain impressions of sensory information (visual, auditory, etc) after the original stimuli have ended. Sensory memory typically only lasts for up to a few seconds. Subcategories include iconic memory (visual), echoic memory (auditory), and haptic memory (touch).Short-Term Memory (STM) or Working Memory: It stores information that we are currently aware of and needed to carry out complex cognitive tasks such as learning and reasoning. Short-term memory is believed to have the capacity of about 7 items (Miller 1956) and lasts for 20-30 seconds.Long-Term Memory (LTM): Long-term memory can store information for a remarkably long time, ranging from a few days to decades, with an essentially unlimited storage capacity. There are two subtypes of LTM:We can roughly consider the following mappings:The external memory can alleviate the restriction of finite attention span.


A standard practice is to save the embedding representation of information into a vector store database that can support fast maximum inner-product search (MIPS). To optimize the retrieval speed, the common choice is the approximate nearest neighbors (ANN) algorithm to return approximately top k nearest neighbors to trade off a little accuracy lost for a huge speedup.A couple common choices of ANN algorithms for fast MIPS:Check more MIPS algorithms and performance comparison in ann-benchmarks.com.Tool use is a remarkable and distinguishing characteristic of human beings. We create, modify and utilize external objects to do things that go beyond our physical and cognitive limits. Equipping LLMs with external tools can significantly extend the model capabilities.MRKL (Karpas et al. 2022), short for “Modular Reasoning, Knowledge and Language”, is a neuro-symbolic architecture for autonomous agents. A MRKL system is proposed to contain a collection of “expert” modules and the general-purpose LLM works as a router to route inquiries to the best suitable expert module. These modules can be neural (e.g. deep learning models) or symbolic (e.g. math calculator, currency converter, weather API).They did an experiment on fine-tuning LLM to call a calculator, using arithmetic as a test case. Their experiments showed that it was harder to solve verbal math problems than explicitly stated math problems because LLMs (7B Jurassic1-large model) failed to extract the right arguments for the basic arithmetic reliably. The results highlight when the external symbolic tools can work reliably, knowing when to and how to use the tools are crucial, determined by the LLM capability.Both TALM (Tool Augmented Language Models; Parisi et al. 2022) and Toolformer (Schick et al. 2023) fine-tune a LM to learn to use external tool APIs. The dataset is expanded based on whether a newly added API call annotation can improve the quality of model outputs. See more details in the “External APIs” section of Prompt Engineering.ChatGPT Plugins and OpenAI API  function calling are good examples of LLMs augmented with tool use capability working in practice. The collection of tool APIs can be provided by other developers (as in Plugins) or self-defined (as in function calls).HuggingGPT (Shen et al. 2023) is a framework to use ChatGPT as the task planner to select models available in HuggingFace platform according to the model descriptions and summarize the response based on the execution results.The system comprises of 4 stages:(1) Task planning: LLM works as the brain and parses the user requests into multiple tasks. There are four attributes associated with each task: task type, ID, dependencies, and arguments. They use few-shot examples to guide LLM to do task parsing and planning.Instruction:(2) Model selection: LLM distributes the tasks to expert models, where the request is framed as a multiple-choice question. LLM is presented with a list of models to choose from. Due to the limited context length, task type based filtration is needed.Instruction:(3) Task execution: Expert models execute on the specific tasks and log results.Instruction:(4) Response generation: LLM receives the execution results and provides summarized results to users.To put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.API-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.In the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:This benchmark evaluates the agent’s tool use capabilities at three levels:ChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:One interesting observation is that while the LLM-based evaluation concluded that GPT-4 and ChemCrow perform nearly equivalently, human evaluations with experts oriented towards the completion and chemical correctness of the solutions showed that ChemCrow outperforms GPT-4 by a large margin. This indicates a potential problem with using LLM to evaluate its own performance on domains that requires deep expertise. The lack of expertise may cause LLMs not knowing its flaws and thus cannot well judge the correctness of task results.Boiko et al. (2023) also looked into LLM-empowered agents for scientific discovery, to handle autonomous design, planning, and performance of complex scientific experiments. This agent can use tools to browse the Internet, read documentation, execute code, call robotics experimentation APIs and leverage other LLMs.For example, when requested to \"develop a novel anticancer drug\", the model came up with the following reasoning steps:They also discussed the risks, especially with illicit drugs and bioweapons. They developed a test set containing a list of known chemical weapon agents and asked the agent to synthesize them. 4 out of 11 requests (36%) were accepted to obtain a synthesis solution and the agent attempted to consult documentation to execute the procedure. 7 out of 11 were rejected and among these 7 rejected cases, 5 happened after a Web search while 2 were rejected based on prompt only.Generative Agents (Park, et al. 2023) is super fun experiment where 25 virtual characters, each controlled by a LLM-powered agent, are living and interacting in a sandbox environment, inspired by The Sims. Generative agents create believable simulacra of human behavior for interactive applications.The design of generative agents combines LLM with memory, planning and reflection mechanisms to enable agents to behave conditioned on past experience, as well as to interact with other agents.This fun simulation results in emergent social behavior, such as information diffusion, relationship memory (e.g. two agents continuing the conversation topic) and coordination of social events (e.g. host a party and invite many others).AutoGPT has drawn a lot of attention into the possibility of setting up autonomous agents with LLM as the main controller. It has quite a lot of reliability issues given the natural language interface, but nevertheless a cool proof-of-concept demo. A lot of code in AutoGPT is about format parsing.Here is the system message used by AutoGPT, where {{...}} are user inputs:GPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.Here are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.Then after these clarification, the agent moved into the code writing mode with a different system message.\n",


"System message:Think step by step and reason yourself to the right decisions to make sure we get it right.\n",
            "You will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.Then you will output the content of each file including ALL code.\n",
            "Each file must strictly follow a markdown code block format, where the following tokens must be replaced such that\n",
            "FILENAME is the lowercase file name including the file extension,\n",
            "LANG is the markup code block language for the code’s language, and CODE is the code:FILENAMEYou will start with the “entrypoint” file, then go to the ones that are imported by that file, and so on.\n",
            "Please note that the code should be fully functional. No placeholders.Follow a language and framework appropriate best practice file naming convention.\n",
            "Make sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other.\n",
            "Ensure to implement all code, if you are unsure, write a plausible implementation.\n",
            "Include module dependency or package manager dependency definition file.\n",
            "Before you finish, double check that all parts of the architecture is present in the files.Useful to know:\n",
            "You almost always put different classes in different files.\n",
            "For Python, you always create an appropriate requirements.txt file.\n",
            "For NodeJS, you always create an appropriate package.json file.\n",
            "You always add a comment briefly describing the purpose of the function definition.\n",
            "You try to add comments explaining very complex bits of logic.\n",
            "You always follow the best practices for the requested languages in terms of describing the code written as a defined\n",
            "package/project.Python toolbelt preferences:Conversatin samples:After going through key ideas and demos of building LLM-centered agents, I start to see a couple common limitations:Finite context length: The restricted context capacity limits the inclusion of historical information, detailed instructions, API call context, and responses. The design of the system has to work with this limited communication bandwidth, while mechanisms like self-reflection to learn from past mistakes would benefit a lot from long or infinite context windows. Although vector stores and retrieval can provide access to a larger knowledge pool, their representation power is not as powerful as full attention.Challenges in long-term planning and task decomposition: Planning over a lengthy history and effectively exploring the solution space remain challenging. LLMs struggle to adjust plans when faced with unexpected errors, making them less robust compared to humans who learn from trial and error.Reliability of natural language interface: Current agent system relies on natural language as an interface between LLMs and external components such as memory and tools. However, the reliability of model outputs is questionable, as LLMs may make formatting errors and occasionally exhibit rebellious behavior (e.g. refuse to follow an instruction). Consequently, much of the agent demo code focuses on parsing model output.Cited as:Weng, Lilian. (Jun 2023). “LLM-powered Autonomous Agents”. Lil’Log. https://lilianweng.github.io/posts/2023-06-23-agent/.Or[1] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022[2] Yao et al. “Tree of Thoughts: Dliberate Problem Solving with Large Language Models.” arXiv preprint arXiv:2305.10601 (2023).[3] Liu et al. “Chain of Hindsight Aligns Language Models with Feedback\n",
            "“ arXiv preprint arXiv:2302.02676 (2023).[4] Liu et al. “LLM+P: Empowering Large Language Models with Optimal Planning Proficiency” arXiv preprint arXiv:2304.11477 (2023).[5] Yao et al. “ReAct: Synergizing reasoning and acting in language models.” ICLR 2023.[6] Google Blog. “Announcing ScaNN: Efficient Vector Similarity Search” July 28, 2020.[7] https://chat.openai.com/share/46ff149e-a4c7-4dd7-a800-fc4a642ea389[8] Shinn & Labash. “Reflexion: an autonomous agent with dynamic memory and self-reflection” arXiv preprint arXiv:2303.11366 (2023).[9] Laskin et al. “In-context Reinforcement Learning with Algorithm Distillation” ICLR 2023.[10] Karpas et al. “MRKL Systems A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning.” arXiv preprint arXiv:2205.00445 (2022).[11] Nakano et al. “Webgpt: Browser-assisted question-answering with human feedback.” arXiv preprint arXiv:2112.09332 (2021).[12] Parisi et al. “TALM: Tool Augmented Language Models”[13] Schick et al. “Toolformer: Language Models Can Teach Themselves to Use Tools.” arXiv preprint arXiv:2302.04761 (2023).[14] Weaviate Blog. Why is Vector Search so fast? Sep 13, 2022.[15] Li et al. “API-Bank: A Benchmark for Tool-Augmented LLMs” arXiv preprint arXiv:2304.08244 (2023).[16] Shen et al. “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace” arXiv preprint arXiv:2303.17580 (2023).[17] Bran et al. “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv preprint arXiv:2304.05376 (2023).[18] Boiko et al. “Emergent autonomous scientific research capabilities of large language models.” arXiv preprint arXiv:2304.05332 (2023).[19] Joon Sung Park, et al. “Generative Agents: Interactive Simulacra of Human Behavior.” arXiv preprint arXiv:2304.03442 (2023).[20] AutoGPT. https://github.com/Significant-Gravitas/Auto-GPT[21] GPT-Engineer. https://github.com/AntonOsika/gpt-engineer\n"
          ]
        }
      ],
      "source": [
        "console.log(loadedDocs[0].pageContent)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Go deeper\n",
        "`DocumentLoader`: Class that loads data from a source as list of Documents. - [Docs](/docs/concepts#document-loaders): Detailed documentation on how to use\n",
        "\n",
        "`DocumentLoaders`. - [Integrations](/docs/integrations/document_loaders/) - [Interface](https:/api.js.langchain.com/classes/langchain.document_loaders_base.BaseDocumentLoader.html): API reference for the base interface."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Indexing: Split\n",
        "Our loaded document is over 42k characters long. This is too long to fit in the context window of many models. Even for those models that could fit the full post in their context window, models can struggle to find information in very long inputs.\n",
        "\n",
        "To handle this we’ll split the `Document` into chunks for embedding and vector storage. This should help us retrieve only the most relevant bits of the blog post at run time.\n",
        "\n",
        "In this case we’ll split our documents into chunks of 1000 characters with 200 characters of overlap between chunks. The overlap helps mitigate the possibility of separating a statement from important context related to it. We use the [RecursiveCharacterTextSplitter](/docs/how_to/recursive_text_splitter/), which will recursively split the document using common separators like new lines until each chunk is the appropriate size. This is the recommended text splitter for generic text use cases."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "const splitter = new RecursiveCharacterTextSplitter({\n",
        "  chunkSize: 1000, chunkOverlap: 200\n",
        "});\n",
        "const allSplits = await splitter.splitDocuments(loadedDocs);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "29\n"
          ]
        }
      ],
      "source": [
        "console.log(allSplits.length);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "996\n"
          ]
        }
      ],
      "source": [


"console.log(allSplits[0].pageContent.length);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  source: 'https://lilianweng.github.io/posts/2023-06-23-agent/',\n",
            "  loc: { lines: { from: 1, to: 1 } }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "allSplits[10].metadata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Go deeper\n",
        "\n",
        "`TextSplitter`: Object that splits a list of `Document`s into smaller chunks. Subclass of `DocumentTransformers`. - Explore `Context-aware splitters`, which keep the location (“context”) of each split in the original `Document`: - [Markdown files](/docs/how_to/code_splitter/#markdown) - [Code](/docs/how_to/code_splitter/) (15+ langs) - [Interface](https://api.js.langchain.com/classes/langchain_textsplitters.TextSplitter.html): API reference for the base interface.\n",
        "\n",
        "`DocumentTransformer`: Object that performs a transformation on a list of `Document`s. - Docs: Detailed documentation on how to use `DocumentTransformer`s - [Integrations](/docs/integrations/document_transformers) - [Interface](https://api.js.langchain.com/classes/langchain_core.documents.BaseDocumentTransformer.html): API reference for the base interface."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Indexing: Store\n",
        "Now we need to index our 28 text chunks so that we can search over them at runtime. The most common way to do this is to embed the contents of each document split and insert these embeddings into a vector database (or vector store). When we want to search over our splits, we take a text search query, embed it, and perform some sort of “similarity” search to identify the stored splits with the most similar embeddings to our query embedding. The simplest similarity measure is cosine similarity — we measure the cosine of the angle between each pair of embeddings (which are high dimensional vectors).\n",
        "\n",
        "We can embed and store all of our document splits in a single command using the [Memory](/docs/integrations/vectorstores/memory) vector store and [OpenAIEmbeddings](/docs/integrations/text_embedding/openai) model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [],
      "source": [
        "import { MemoryVectorStore } from \"langchain/vectorstores/memory\"\n",
        "import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
        "\n",
        "const inMemoryVectorStore = await MemoryVectorStore.fromDocuments(allSplits, new OpenAIEmbeddings());"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Go deeper\n",
        "\n",
        "`Embeddings`: Wrapper around a text embedding model, used for converting text to embeddings. - [Docs](/docs/concepts#embedding-models): Detailed documentation on how to use embeddings. - [Integrations](/docs/integrations/text_embedding): 30+ integrations to choose from. - [Interface](https://api.js.langchain.com/classes/langchain_core.embeddings.Embeddings.html): API reference for the base interface.\n",
        "\n",
        "`VectorStore`: Wrapper around a vector database, used for storing and querying embeddings. - [Docs](/docs/concepts#vectorstores): Detailed documentation on how to use vector stores. - [Integrations](/docs/integrations/vectorstores): 40+ integrations to choose from. - [Interface](https://api.js.langchain.com/classes/langchain_core.vectorstores.VectorStore.html): API reference for the base interface.\n",
        "\n",
        "This completes the **Indexing** portion of the pipeline. At this point we have a query-able vector store containing the chunked contents of our blog post. Given a user question, we should ideally be able to return the snippets of the blog post that answer the question."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Retrieval and Generation: Retrieve\n",
        "\n",
        "Now let’s write the actual application logic. We want to create a simple application that takes a user question, searches for documents relevant to that question, passes the retrieved documents and initial question to a model, and returns an answer.\n",
        "\n",
        "First we need to define our logic for searching over documents. LangChain defines a [Retriever](/docs/concepts#retrievers) interface which wraps an index that can return relevant `Document`s given a string query.\n",
        "\n",
        "The most common type of Retriever is the [VectorStoreRetriever](https://api.js.langchain.com/classes/langchain_core.vectorstores.VectorStoreRetriever.html), which uses the similarity search capabilities of a vector store to facilitate retrieval. Any `VectorStore` can easily be turned into a `Retriever` with `VectorStore.asRetriever()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "const vectorStoreRetriever = inMemoryVectorStore.asRetriever({ k: 6, searchType: \"similarity\" });"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "const retrievedDocuments = await vectorStoreRetriever.invoke(\"What are the approaches to task decomposition?\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "6\n"
          ]
        }
      ],
      "source": [
        "console.log(retrievedDocuments.length);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.Another quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain\n"
          ]
        }
      ],
      "source": [
        "console.log(retrievedDocuments[0].pageContent);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Go deeper\n",
        "\n",
        "Vector stores are commonly used for retrieval, but there are other ways to do retrieval, too.\n",
        "\n",
        "`Retriever`: An object that returns `Document`s given a text query - [Docs](/docs/concepts#retrievers): Further documentation on the interface and built-in retrieval techniques. Some of which include: - `MultiQueryRetriever` [generates variants of the input question](/docs/how_to/multiple_queries/) to improve retrieval hit rate. - `MultiVectorRetriever` (diagram below) instead generates variants of the embeddings, also in order to improve retrieval hit rate. - Max marginal relevance selects for relevance and diversity among the retrieved documents to avoid passing in duplicate context. - Documents can be filtered during vector store retrieval using metadata filters. - Integrations: Integrations with retrieval services. - Interface: API reference for the base interface."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Retrieval and Generation: Generate\n",
        "\n",
        "Let’s put it all together into a chain that takes a question, retrieves relevant documents, constructs a prompt, passes that to a model, and parses the output.\n",
        "\n",


# Build a Question/Answering system over SQL data

:::info Prerequisites

This guide assumes familiarity with the following concepts:

- [Chaining runnables](/docs/how_to/sequence/)
- [Chat models](/docs/concepts/#chat-models)
- [Tools](/docs/concepts/#tools)
- [Agents](/docs/concepts/#agents)

:::

In this guide we'll go over the basic ways to create a Q&A chain and agent over a SQL database.
These systems will allow us to ask a question about the data in a SQL database and get back a natural language answer.
The main difference between the two is that our agent can query the database in a loop as many time as it needs to answer the question.

## ⚠ Security note ⚠

Building Q&A systems of SQL databases can require executing model-generated SQL queries. There are inherent risks in doing this.
Make sure that your database connection permissions are always scoped as narrowly as possible for your chain/agent's needs.
This will mitigate though not eliminate the risks of building a model-driven system. For more on general security best practices, see [here](/docs/security).

## Architecture

At a high-level, the steps of most SQL chain and agent are:

1. **Convert question to SQL query**: Model converts user input to a SQL query.
2. **Execute SQL query**: Execute the SQL query
3. **Answer the question**: Model responds to user input using the query results.

![SQL Use Case Diagram](/img/sql_usecase.png)

## Setup

First, get required packages and set environment variables:

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm i langchain @langchain/community @langchain/openai @langchain/core
```

We default to OpenAI models in this guide.

```bash
export OPENAI_API_KEY=<your key>

# Uncomment the below to use LangSmith. Not required, but recommended for debugging and observability.
# export LANGCHAIN_API_KEY=<your key>
# export LANGCHAIN_TRACING_V2=true

# Reduce tracing latency if you are not in a serverless environment
# export LANGCHAIN_CALLBACKS_BACKGROUND=true
```

import CodeBlock from "@theme/CodeBlock";
import DbCheck from "@examples/use_cases/sql/db_check.ts";

<CodeBlock language="typescript">{DbCheck}</CodeBlock>

Great! We've got a SQL database that we can query. Now let's try hooking it up to an LLM.

## Chain

Let's create a simple chain that takes a question, turns it into a SQL query, executes the query, and uses the result to answer the original question.

### Convert question to SQL query

The first step in a SQL chain or agent is to take the user input and convert it to a SQL query. LangChain comes with a built-in chain for this: [`createSqlQueryChain`](https://api.js.langchain.com/functions/langchain.chains_sql_db.createSqlQueryChain.html)

import QuickstartChainExample from "@examples/use_cases/sql/quickstart_chain.ts";

<CodeBlock language="typescript">{QuickstartChainExample}</CodeBlock>

We can look at the [LangSmith trace](https://smith.langchain.com/public/6d8f0213-9f02-498e-aeb2-ec774e324e2c/r) to get a better understanding of what this chain is doing.
We can also inspect the chain directly for its prompts. Looking at the prompt (below), we can see that it is:

- Dialect-specific. In this case it references SQLite explicitly.
- Has definitions for all the available tables.
- Has three examples rows for each table.

This technique is inspired by papers like [this](https://arxiv.org/pdf/2204.00498.pdf), which suggest showing examples rows and being explicit about tables improves performance.
We can also inspect the full prompt via the LangSmith trace:

![Chain Prompt](/img/sql_quickstart_langsmith_prompt.png)

### Execute SQL query

Now that we've generated a SQL query, we'll want to execute it.
This is the most dangerous part of creating a SQL chain. Consider carefully if it is OK to run automated queries over your data.
Minimize the database connection permissions as much as possible.
Consider adding a human approval step to you chains before query execution (see below).

We can use the [`QuerySqlTool`](https://api.js.langchain.com/classes/langchain.tools_sql.QuerySqlTool.html) to easily add query execution to our chain:

import QuickstartExecuteExample from "@examples/use_cases/sql/quickstart_execute_sql.ts";

<CodeBlock language="typescript">{QuickstartExecuteExample}</CodeBlock>

:::tip
See a LangSmith trace of the chain above [here](https://smith.langchain.com/public/3cbcf6f2-a55b-4701-a2e3-9928e4747328/r).
:::

### Answer the question

Now that we have a way to automatically generate and execute queries, we just need to combine the original question and SQL query result to generate a final answer.
We can do this by passing question and result to the LLM once more:

import QuickstartAnswerExample from "@examples/use_cases/sql/quickstart_answer_question.ts";

<CodeBlock language="typescript">{QuickstartAnswerExample}</CodeBlock>

:::tip
See a LangSmith trace of the chain above [here](https://smith.langchain.com/public/d130ce1f-1fce-4192-921e-4b522884ec1a/r).
:::

### Next steps

For more complex query-generation, we may want to create few-shot prompts or add query-checking steps. For advanced techniques like this and more check out:

- [Prompting strategies](/docs/how_to/sql_prompting): Advanced prompt engineering techniques.
- [Query checking](/docs/how_to/sql_query_checking): Add query validation and error handling.
- [Large databases](/docs/how_to/sql_large_db): Techniques for working with large databases.

## Agents

LangChain offers a number of tools and functions that allow you to create SQL Agents which can provide a more flexible way of interacting with SQL databases. The main advantages of using SQL Agents are:

- It can answer questions based on the databases' schema as well as on the databases' content (like describing a specific table).
- It can recover from errors by running a generated query, catching the traceback and regenerating it correctly.
- It can answer questions that require multiple dependent queries.
- It will save tokens by only considering the schema from relevant tables.
- To initialize the agent, we use [`createOpenAIToolsAgent`](https://api.js.langchain.com/functions/langchain.agents.createOpenAIToolsAgent.html) function.
  This agent contains the [`SqlToolkit`](https://api.js.langchain.com/classes/langchain.agents_toolkits_sql.SqlToolkit.html) which contains tools to:
- Create and execute queries
- Check query syntax
- Retrieve table descriptions
- … and more


import type * as tiktoken from "js-tiktoken";
import { Document, BaseDocumentTransformer } from "@langchain/core/documents";
import { getEncoding } from "@langchain/core/utils/tiktoken";

export interface TextSplitterParams {
  chunkSize: number;
  chunkOverlap: number;
  keepSeparator: boolean;
  lengthFunction?:
    | ((text: string) => number)
    | ((text: string) => Promise<number>);
}

export type TextSplitterChunkHeaderOptions = {
  chunkHeader?: string;
  chunkOverlapHeader?: string;
  appendChunkOverlapHeader?: boolean;
};

export abstract class TextSplitter
  extends BaseDocumentTransformer
  implements TextSplitterParams
{
  lc_namespace = ["langchain", "document_transformers", "text_splitters"];

  chunkSize = 1000;

  chunkOverlap = 200;

  keepSeparator = false;

  lengthFunction:
    | ((text: string) => number)
    | ((text: string) => Promise<number>);

  constructor(fields?: Partial<TextSplitterParams>) {
    super(fields);
    this.chunkSize = fields?.chunkSize ?? this.chunkSize;
    this.chunkOverlap = fields?.chunkOverlap ?? this.chunkOverlap;
    this.keepSeparator = fields?.keepSeparator ?? this.keepSeparator;
    this.lengthFunction =
      fields?.lengthFunction ?? ((text: string) => text.length);
    if (this.chunkOverlap >= this.chunkSize) {
      throw new Error("Cannot have chunkOverlap >= chunkSize");
    }
  }

  async transformDocuments(
    documents: Document[],
    chunkHeaderOptions: TextSplitterChunkHeaderOptions = {}
  ): Promise<Document[]> {
    return this.splitDocuments(documents, chunkHeaderOptions);
  }

  abstract splitText(text: string): Promise<string[]>;

  protected splitOnSeparator(text: string, separator: string): string[] {
    let splits;
    if (separator) {
      if (this.keepSeparator) {
        const regexEscapedSeparator = separator.replace(
          /[/\-\\^$*+?.()|[\]{}]/g,
          "\\$&"
        );
        splits = text.split(new RegExp(`(?=${regexEscapedSeparator})`));
      } else {
        splits = text.split(separator);
      }
    } else {
      splits = text.split("");
    }
    return splits.filter((s) => s !== "");
  }

  async createDocuments(
    texts: string[],
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    metadatas: Record<string, any>[] = [],
    chunkHeaderOptions: TextSplitterChunkHeaderOptions = {}
  ): Promise<Document[]> {
    // if no metadata is provided, we create an empty one for each text
    // eslint-disable-next-line @typescript-eslint/no-explicit-any
    const _metadatas: Record<string, any>[] =
      metadatas.length > 0
        ? metadatas
        : [...Array(texts.length)].map(() => ({}));
    const {
      chunkHeader = "",
      chunkOverlapHeader = "(cont'd) ",
      appendChunkOverlapHeader = false,
    } = chunkHeaderOptions;
    const documents = new Array<Document>();
    for (let i = 0; i < texts.length; i += 1) {
      const text = texts[i];
      let lineCounterIndex = 1;
      let prevChunk = null;
      let indexPrevChunk = -1;
      for (const chunk of await this.splitText(text)) {
        let pageContent = chunkHeader;

        // we need to count the \n that are in the text before getting removed by the splitting
        const indexChunk = text.indexOf(chunk, indexPrevChunk + 1);
        if (prevChunk === null) {
          const newLinesBeforeFirstChunk = this.numberOfNewLines(
            text,
            0,
            indexChunk
          );
          lineCounterIndex += newLinesBeforeFirstChunk;
        } else {
          const indexEndPrevChunk =
            indexPrevChunk + (await this.lengthFunction(prevChunk));
          if (indexEndPrevChunk < indexChunk) {
            const numberOfIntermediateNewLines = this.numberOfNewLines(
              text,
              indexEndPrevChunk,
              indexChunk
            );
            lineCounterIndex += numberOfIntermediateNewLines;
          } else if (indexEndPrevChunk > indexChunk) {
            const numberOfIntermediateNewLines = this.numberOfNewLines(
              text,
              indexChunk,
              indexEndPrevChunk
            );
            lineCounterIndex -= numberOfIntermediateNewLines;
          }
          if (appendChunkOverlapHeader) {
            pageContent += chunkOverlapHeader;
          }
        }
        const newLinesCount = this.numberOfNewLines(chunk);

        const loc =
          _metadatas[i].loc && typeof _metadatas[i].loc === "object"
            ? { ..._metadatas[i].loc }
            : {};
        loc.lines = {
          from: lineCounterIndex,
          to: lineCounterIndex + newLinesCount,
        };
        const metadataWithLinesNumber = {
          ..._metadatas[i],
          loc,
        };

        pageContent += chunk;
        documents.push(
          new Document({
            pageContent,
            metadata: metadataWithLinesNumber,
          })
        );
        lineCounterIndex += newLinesCount;
        prevChunk = chunk;
        indexPrevChunk = indexChunk;
      }
    }
    return documents;
  }

  private numberOfNewLines(text: string, start?: number, end?: number) {
    const textSection = text.slice(start, end);
    return (textSection.match(/\n/g) || []).length;
  }

  async splitDocuments(
    documents: Document[],
    chunkHeaderOptions: TextSplitterChunkHeaderOptions = {}
  ): Promise<Document[]> {
    const selectedDocuments = documents.filter(
      (doc) => doc.pageContent !== undefined
    );
    const texts = selectedDocuments.map((doc) => doc.pageContent);
    const metadatas = selectedDocuments.map((doc) => doc.metadata);
    return this.createDocuments(texts, metadatas, chunkHeaderOptions);
  }

  private joinDocs(docs: string[], separator: string): string | null {
    const text = docs.join(separator).trim();
    return text === "" ? null : text;
  }

  async mergeSplits(splits: string[], separator: string): Promise<string[]> {
    const docs: string[] = [];
    const currentDoc: string[] = [];
    let total = 0;
    for (const d of splits) {
      const _len = await this.lengthFunction(d);
      if (
        total + _len + currentDoc.length * separator.length >
        this.chunkSize
      ) {
        if (total > this.chunkSize) {
          console.warn(
            `Created a chunk of size ${total}, +
which is longer than the specified ${this.chunkSize}`
          );
        }
        if (currentDoc.length > 0) {
          const doc = this.joinDocs(currentDoc, separator);
          if (doc !== null) {
            docs.push(doc);
          }
          // Keep on popping if:
          // - we have a larger chunk than in the chunk overlap
          // - or if we still have any chunks and the length is long
          while (
            total > this.chunkOverlap ||
            (total + _len + currentDoc.length * separator.length >
              this.chunkSize &&
              total > 0)
          ) {
            total -= await this.lengthFunction(currentDoc[0]);
            currentDoc.shift();
          }
        }
      }
      currentDoc.push(d);
      total += _len;
    }
    const doc = this.joinDocs(currentDoc, separator);
    if (doc !== null) {
      docs.push(doc);
    }
    return docs;
  }
}

export interface CharacterTextSplitterParams extends TextSplitterParams {
  separator: string;
}

export class CharacterTextSplitter
  extends TextSplitter
  implements CharacterTextSplitterParams
{
  static lc_name() {
    return "CharacterTextSplitter";
  }

  separator = "\n\n";

  constructor(fields?: Partial<CharacterTextSplitterParams>) {
    super(fields);
    this.separator = fields?.separator ?? this.separator;
  }

  async splitText(text: string): Promise<string[]> {
    // First we naively split the large input into a bunch of smaller ones.
    const splits = this.splitOnSeparator(text, this.separator);
    return this.mergeSplits(splits, this.keepSeparator ? "" : this.separator);
  }
}

export interface RecursiveCharacterTextSplitterParams
  extends TextSplitterParams {
  separators: string[];
}

export const SupportedTextSplitterLanguages = [
  "cpp",
  "go",
  "java",
  "js",
  "php",
  "proto",
  "python",
  "rst",
  "ruby",
  "rust",
  "scala",
  "swift",
  "markdown",
  "latex",
  "html",
  "sol",
] as const;

export type SupportedTextSplitterLanguage =
  (typeof SupportedTextSplitterLanguages)[number];


export class RecursiveCharacterTextSplitter
  extends TextSplitter
  implements RecursiveCharacterTextSplitterParams
{
  static lc_name() {
    return "RecursiveCharacterTextSplitter";
  }

  separators: string[] = ["\n\n", "\n", " ", ""];

  constructor(fields?: Partial<RecursiveCharacterTextSplitterParams>) {
    super(fields);
    this.separators = fields?.separators ?? this.separators;
    this.keepSeparator = fields?.keepSeparator ?? true;
  }

  private async _splitText(text: string, separators: string[]) {
    const finalChunks: string[] = [];

    // Get appropriate separator to use
    let separator: string = separators[separators.length - 1];
    let newSeparators;
    for (let i = 0; i < separators.length; i += 1) {
      const s = separators[i];
      if (s === "") {
        separator = s;
        break;
      }
      if (text.includes(s)) {
        separator = s;
        newSeparators = separators.slice(i + 1);
        break;
      }
    }

    // Now that we have the separator, split the text
    const splits = this.splitOnSeparator(text, separator);

    // Now go merging things, recursively splitting longer texts.
    let goodSplits: string[] = [];
    const _separator = this.keepSeparator ? "" : separator;
    for (const s of splits) {
      if ((await this.lengthFunction(s)) < this.chunkSize) {
        goodSplits.push(s);
      } else {
        if (goodSplits.length) {
          const mergedText = await this.mergeSplits(goodSplits, _separator);
          finalChunks.push(...mergedText);
          goodSplits = [];
        }
        if (!newSeparators) {
          finalChunks.push(s);
        } else {
          const otherInfo = await this._splitText(s, newSeparators);
          finalChunks.push(...otherInfo);
        }
      }
    }
    if (goodSplits.length) {
      const mergedText = await this.mergeSplits(goodSplits, _separator);
      finalChunks.push(...mergedText);
    }
    return finalChunks;
  }

  async splitText(text: string): Promise<string[]> {
    return this._splitText(text, this.separators);
  }

  static fromLanguage(
    language: SupportedTextSplitterLanguage,
    options?: Partial<RecursiveCharacterTextSplitterParams>
  ) {
    return new RecursiveCharacterTextSplitter({
      ...options,
      separators:
        RecursiveCharacterTextSplitter.getSeparatorsForLanguage(language),
    });
  }