"For the retriever, we will use [WebBaseLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) to load the content of a web page. Here we instantiate a `InMemoryVectorStore` vectorstore and then use its [.as_retriever](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.VectorStore.html#langchain_core.vectorstores.VectorStore.as_retriever) method to build a retriever that can be incorporated into [LCEL](/docs/concepts/#langchain-expression-language) chains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "820244ae-74b4-4593-b392-822979dd91b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "USER_AGENT environment variable not set, consider setting it to identify your requests.\n"
     ]
    }
   ],
   "source": [
    "import bs4\n",
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "loader = WebBaseLoader(\n",
    "    web_paths=(\"https://lilianweng.github.io/posts/2023-06-23-agent/\",),\n",
    "    bs_kwargs=dict(\n",
    "        parse_only=bs4.SoupStrainer(\n",
    "            class_=(\"post-content\", \"post-title\", \"post-header\")\n",
    "        )\n",
    "    ),\n",
    ")\n",
    "docs = loader.load()\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\n",
    "splits = text_splitter.split_documents(docs)\n",
    "vectorstore = InMemoryVectorStore(embedding=OpenAIEmbeddings())\n",
    "vectorstore.add_documents(splits)\n",
    "retriever = vectorstore.as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776ae958-cbdc-4471-8669-c6087436f0b5",
   "metadata": {},
   "source": [
    "### Prompt\n",
    "\n",
    "We'll use a prompt that includes a `MessagesPlaceholder` variable under the name \"chat_history\". This allows us to pass in a list of Messages to the prompt using the \"chat_history\" input key, and these messages will be inserted after the system message and before the human message containing the latest question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2b685428-8b82-4af1-be4f-7232c5d55b73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_history_aware_retriever\n",
    "from langchain_core.prompts import MessagesPlaceholder\n",
    "\n",
    "contextualize_q_system_prompt = (\n",
    "    \"Given a chat history and the latest user question \"\n",
    "    \"which might reference context in the chat history, \"\n",
    "    \"formulate a standalone question which can be understood \"\n",
    "    \"without the chat history. Do NOT answer the question, \"\n",
    "    \"just reformulate it if needed and otherwise return it as is.\"\n",
    ")\n",
    "\n",
    "contextualize_q_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", contextualize_q_system_prompt),\n",
    "        MessagesPlaceholder(\"chat_history\"),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d2a692e-a019-4515-9625-5b0530c3c9af",
   "metadata": {},
   "source": [
    "### Assembling the chain\n",
    "\n",
    "We can then instantiate the history-aware retriever:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4c4b1695-6217-4ee8-abaf-7cc26366d988",
   "metadata": {},
   "outputs": [],
   "source": [
    "history_aware_retriever = create_history_aware_retriever(\n",
    "    llm, retriever, contextualize_q_prompt\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a47168-4a1f-4e39-bd2d-d5b03609a243",
   "metadata": {},
   "source": [
    "This chain prepends a rephrasing of the input query to our retriever, so that the retrieval incorporates the context of the conversation.\n",
    "\n",
    "Now we can build our full QA chain.\n",
    "\n",
    "As in the [RAG tutorial](/docs/tutorials/rag), we will use [create_stuff_documents_chain](https://python.langchain.com/api_reference/langchain/chains/langchain.chains.combine_documents.stuff.create_stuff_documents_chain.html) to generate a `question_answer_chain`, with input keys `context`, `chat_history`, and `input`-- it accepts the retrieved context alongside the conversation history and query to generate an answer.\n",
    "\n",
    "We build our final `rag_chain` with [create_retrieval_chain](https://python.langchain.com/api_reference/langchain/chains/langchain.chains.retrieval.create_retrieval_chain.html). This chain applies the `history_aware_retriever` and `question_answer_chain` in sequence, retaining intermediate outputs such as the retrieved context for convenience. It has input keys `input` and `chat_history`, and includes `input`, `chat_history`, `context`, and `answer` in its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "afef4385-f571-4874-8f52-3d475642f579",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = (\n",
    "    \"You are an assistant for question-answering tasks. \"\n",
    "    \"Use the following pieces of retrieved context to answer \"\n",
    "    \"the question. If you don't know the answer, say that you \"\n",
    "    \"don't know. Use three sentences maximum and keep the \"\n",
    "    \"answer concise.\"\n",
    "    \"\\n\\n\"\n",
    "    \"{context}\"\n",
    ")\n",
    "qa_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system_prompt),\n",
    "        MessagesPlaceholder(\"chat_history\"),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)\n",
    "rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53a662c2-f38b-45f9-95c4-66de15637614",
   "metadata": {},
   "source": [
    "### Stateful Management of chat history\n",
    "\n",
    "We have added application logic for incorporating chat history, but we are still manually plumbing it through our application. In production, the Q&A application we usually persist the chat history into a database, and be able to read and update it appropriately.\n",
    "\n",
    "[LangGraph](https://langchain-ai.github.io/langgraph/) implements a built-in [persistence layer](https://langchain-ai.github.io/langgraph/concepts/persistence/), making it ideal for chat applications that support multiple conversational turns.\n",
    "\n",
    "Wrapping our chat model in a minimal LangGraph application allows us to automatically persist the message history, simplifying the development of multi-turn applications.\n",
    "\n",


"LangGraph comes with a simple [in-memory checkpointer](https://langchain-ai.github.io/langgraph/reference/checkpoints/#memorysaver), which we use below. See its documentation for more detail, including how to use different persistence backends (e.g., SQLite or Postgres).\n",
    "\n",
    "For a detailed walkthrough of how to manage message history, head to the How to add message history (memory) guide."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9c3fb176-8d6a-4dc7-8408-6a22c5f7cc72",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Sequence\n",
    "\n",
    "from langchain_core.messages import AIMessage, BaseMessage, HumanMessage\n",
    "from langgraph.checkpoint.memory import MemorySaver\n",
    "from langgraph.graph import START, StateGraph\n",
    "from langgraph.graph.message import add_messages\n",
    "from typing_extensions import Annotated, TypedDict\n",
    "\n",
    "\n",
    "# We define a dict representing the state of the application.\n",
    "# This state has the same input and output keys as `rag_chain`.\n",
    "class State(TypedDict):\n",
    "    input: str\n",
    "    chat_history: Annotated[Sequence[BaseMessage], add_messages]\n",
    "    context: str\n",
    "    answer: str\n",
    "\n",
    "\n",
    "# We then define a simple node that runs the `rag_chain`.\n",
    "# The `return` values of the node update the graph state, so here we just\n",
    "# update the chat history with the input message and response.\n",
    "def call_model(state: State):\n",
    "    response = rag_chain.invoke(state)\n",
    "    return {\n",
    "        \"chat_history\": [\n",
    "            HumanMessage(state[\"input\"]),\n",
    "            AIMessage(response[\"answer\"]),\n",
    "        ],\n",
    "        \"context\": response[\"context\"],\n",
    "        \"answer\": response[\"answer\"],\n",
    "    }\n",
    "\n",
    "\n",
    "# Our graph consists only of one node:\n",
    "workflow = StateGraph(state_schema=State)\n",
    "workflow.add_edge(START, \"model\")\n",
    "workflow.add_node(\"model\", call_model)\n",
    "\n",
    "# Finally, we compile the graph with a checkpointer object.\n",
    "# This persists the state, in this case in memory.\n",
    "memory = MemorySaver()\n",
    "app = workflow.compile(checkpointer=memory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1046c92f-21b3-4214-907d-92878d8cba23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. This process helps agents or models tackle difficult tasks by dividing them into more manageable subtasks. Task decomposition can be achieved through methods like Chain of Thought (CoT) or Tree of Thoughts, which guide the agent in thinking step by step or exploring multiple reasoning possibilities at each step.\n"
     ]
    }
   ],
   "source": [
    "config = {\"configurable\": {\"thread_id\": \"abc123\"}}\n",
    "\n",
    "result = app.invoke(\n",
    "    {\"input\": \"What is Task Decomposition?\"},\n",
    "    config=config,\n",
    ")\n",
    "print(result[\"answer\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0e89c75f-7ad7-4331-a2fe-57579eb8f840",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "One way of task decomposition is by using Large Language Models (LLMs) with simple prompting, such as providing instructions like \"Steps for XYZ\" or asking about subgoals for achieving a specific task. This method leverages the power of LLMs to break down tasks into smaller components for easier handling. Additionally, task decomposition can also be done using task-specific instructions tailored to the nature of the task, like requesting a story outline for writing a novel.\n"
     ]
    }
   ],
   "source": [
    "result = app.invoke(\n",
    "    {\"input\": \"What is one way of doing it?\"},\n",
    "    config=config,\n",
    ")\n",
    "print(result[\"answer\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab59258-84bc-4904-880e-2ebfebbca563",
   "metadata": {},
   "source": [
    "The conversation history can be inspected via the state of the application:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7686b874-3a85-499f-82b5-28a85c4c768c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================\u001b[1m Human Message \u001b[0m=================================\n",
      "\n",
      "What is Task Decomposition?\n",
      "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
      "\n",
      "Task decomposition is a technique used to break down complex tasks into smaller and simpler steps. This process helps agents or models tackle difficult tasks by dividing them into more manageable subtasks. Task decomposition can be achieved through methods like Chain of Thought (CoT) or Tree of Thoughts, which guide the agent in thinking step by step or exploring multiple reasoning possibilities at each step.\n",
      "================================\u001b[1m Human Message \u001b[0m=================================\n",
      "\n",
      "What is one way of doing it?\n",
      "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
      "\n",
      "One way of task decomposition is by using Large Language Models (LLMs) with simple prompting, such as providing instructions like \"Steps for XYZ\" or asking about subgoals for achieving a specific task. This method leverages the power of LLMs to break down tasks into smaller components for easier handling. Additionally, task decomposition can also be done using task-specific instructions tailored to the nature of the task, like requesting a story outline for writing a novel.\n"
     ]
    }
   ],
   "source": [
    "chat_history = app.get_state(config).values[\"chat_history\"]\n",
    "for message in chat_history:\n",
    "    message.pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ab1ded4-76d9-453f-9b9b-db9a4560c737",
   "metadata": {},
   "source": [
    "### Tying it together"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a08a5ea-df5b-4547-93c6-2a3940dd5c3e",
   "metadata": {},
   "source": [
    "![](../../static/img/conversational_retrieval_chain.png)\n",
    "\n",
    "For convenience, we tie together all of the necessary steps in a single code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "71c32048-1a41-465f-a9e2-c4affc332fd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Sequence\n",
    "\n",
    "import bs4\n",
    "from langchain.chains import create_history_aware_retriever, create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_core.messages import AIMessage, BaseMessage, HumanMessage\n",
    "from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
    "from langchain_core.runnables.history import RunnableWithMessageHistory\n",
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",


"To explore different types of retrievers and retrieval strategies, visit the [retrievers](/docs/how_to#retrievers) section of the how-to guides.\n",
    "\n",
    "For a detailed walkthrough of LangChain's conversation memory abstractions, visit the [How to add message history (memory)](/docs/how_to/message_history) LCEL page.\n",
    "\n",
    "To learn more about agents, head to the [Agents Modules](/docs/tutorials/agents)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


"1. We will add similarity scores to the metadata of the corresponding \"sub-documents\" using the `similarity_search_with_score` method of the underlying vector store as above;\n",
    "2. We will include a list of these sub-documents in the metadata of the retrieved parent document. This surfaces what snippets of text were identified by the retrieval, together with their corresponding similarity scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1de61de7-1b58-41d6-9dea-939fef7d741d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "from langchain.retrievers import MultiVectorRetriever\n",
    "from langchain_core.callbacks import CallbackManagerForRetrieverRun\n",
    "\n",
    "\n",
    "class CustomMultiVectorRetriever(MultiVectorRetriever):\n",
    "    def _get_relevant_documents(\n",
    "        self, query: str, *, run_manager: CallbackManagerForRetrieverRun\n",
    "    ) -> List[Document]:\n",
    "        \"\"\"Get documents relevant to a query.\n",
    "        Args:\n",
    "            query: String to find relevant documents for\n",
    "            run_manager: The callbacks handler to use\n",
    "        Returns:\n",
    "            List of relevant documents\n",
    "        \"\"\"\n",
    "        results = self.vectorstore.similarity_search_with_score(\n",
    "            query, **self.search_kwargs\n",
    "        )\n",
    "\n",
    "        # Map doc_ids to list of sub-documents, adding scores to metadata\n",
    "        id_to_doc = defaultdict(list)\n",
    "        for doc, score in results:\n",
    "            doc_id = doc.metadata.get(\"doc_id\")\n",
    "            if doc_id:\n",
    "                doc.metadata[\"score\"] = score\n",
    "                id_to_doc[doc_id].append(doc)\n",
    "\n",
    "        # Fetch documents corresponding to doc_ids, retaining sub_docs in metadata\n",
    "        docs = []\n",
    "        for _id, sub_docs in id_to_doc.items():\n",
    "            docstore_docs = self.docstore.mget([_id])\n",
    "            if docstore_docs:\n",
    "                if doc := docstore_docs[0]:\n",
    "                    doc.metadata[\"sub_docs\"] = sub_docs\n",
    "                    docs.append(doc)\n",
    "\n",
    "        return docs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7af27b38-631c-463f-9d66-bcc985f06a4f",
   "metadata": {},
   "source": [
    "Invoking this retriever, we can see that it identifies the correct parent document, including the relevant snippet from the sub-document with similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dc42a1be-22e1-4ade-b1bd-bafb85f2424f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='fake whole document 1', metadata={'sub_docs': [Document(page_content='A snippet from a larger document discussing cats.', metadata={'doc_id': 'fake_id_1', 'score': 0.831276655})]})]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever = CustomMultiVectorRetriever(vectorstore=vectorstore, docstore=docstore)\n",
    "\n",
    "retriever.invoke(\"cat\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1b79ff35-50a3-40cd-86d9-703f1f8cd2c5",
   "metadata": {},
   "source": [
    "# How to get a RAG application to add citations\n",
    "\n",
    "This guide reviews methods to get a model to cite which parts of the source documents it referenced in generating its response.\n",
    "\n",
    "We will cover five methods:\n",
    "\n",
    "1. Using tool-calling to cite document IDs;\n",
    "2. Using tool-calling to cite documents IDs and provide text snippets;\n",
    "3. Direct prompting;\n",
    "4. Retrieval post-processing (i.e., compressing the retrieved context to make it more relevant);\n",
    "5. Generation post-processing (i.e., issuing a second LLM call to annotate a generated answer with citations).\n",
    "\n",
    "We generally suggest using the first item of the list that works for your use-case. That is, if your model supports tool-calling, try methods 1 or 2; otherwise, or if those fail, advance down the list.\n",
    "\n",
    "Let's first create a simple RAG chain. To start we'll just retrieve from Wikipedia using the [WikipediaRetriever](https://python.langchain.com/api_reference/community/retrievers/langchain_community.retrievers.wikipedia.WikipediaRetriever.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a70c423-f61f-4230-b70a-d3605b31afab",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First we'll need to install some dependencies and set environment vars for the models we'll be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f1d26ded-e8d5-4f80-86b9-26d464869175",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain langchain-openai langchain-anthropic langchain-community wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8732a85a-dd1a-483c-8da7-a81251276aa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()\n",
    "os.environ[\"ANTHROPIC_API_KEY\"] = getpass.getpass()\n",
    "\n",
    "# Uncomment if you want to log to LangSmith\n",
    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\n",
    "# os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a4401f-7feb-4bd9-9409-77c3859c4292",
   "metadata": {},
   "source": [
    "Let's first select a LLM:\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs customVarName=\"llm\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dd00165d-0b32-466d-8f75-ec26326a9e36",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4e17c3f6-8ce6-4767-b615-50a57c84c7b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================\u001b[1m System Message \u001b[0m================================\n",
      "\n",
      "You're a helpful AI assistant. Given a user question and some Wikipedia article snippets, answer the user question. If none of the articles answer the question, just say you don't know.\n",
      "\n",
      "Here are the Wikipedia articles: \u001b[33;1m\u001b[1;3m{context}\u001b[0m\n",
      "\n",
      "================================\u001b[1m Human Message \u001b[0m=================================\n",
      "\n",
      "\u001b[33;1m\u001b[1;3m{input}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.retrievers import WikipediaRetriever\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "system_prompt = (\n",
    "    \"You're a helpful AI assistant. Given a user question \"\n",
    "    \"and some Wikipedia article snippets, answer the user \"\n",
    "    \"question. If none of the articles answer the question, \"\n",
    "    \"just say you don't know.\"\n",
    "    \"\\n\\nHere are the Wikipedia articles: \"\n",
    "    \"{context}\"\n",
    ")\n",
    "\n",
    "retriever = WikipediaRetriever(top_k_results=6, doc_content_chars_max=2000)\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system_prompt),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "prompt.pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c89e2045-9244-43e6-bf3f-59af22658529",
   "metadata": {},
   "source": [
    "Now that we've got a model, retriver and prompt, let's chain them all together. We'll need to add some logic for formatting our retrieved Documents to a string that can be passed to our prompt. Following the how-to guide on [adding citations](/docs/how_to/qa_citations) to a RAG application, we'll make it so our chain returns both the answer and the retrieved Documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4cd55e1c-a6b7-44b7-9dde-5f42abe714ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from langchain_core.documents import Document\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "\n",
    "\n",
    "def format_docs(docs: List[Document]):\n",
    "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
    "\n",
    "\n",
    "rag_chain_from_docs = (\n",
    "    RunnablePassthrough.assign(context=(lambda x: format_docs(x[\"context\"])))\n",
    "    | prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "retrieve_docs = (lambda x: x[\"input\"]) | retriever\n",
    "\n",
    "chain = RunnablePassthrough.assign(context=retrieve_docs).assign(\n",
    "    answer=rag_chain_from_docs\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "42b28717-d34c-42de-b923-155ac60529a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = chain.invoke({\"input\": \"How fast are cheetahs?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8b20cf8e-dccd-45d1-aef0-25f1ad1aca6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['input', 'context', 'answer'])\n"
     ]
    }
   ],
   "source": [
    "print(result.keys())"
   ]


"Another approach is to post-process our retrieved documents to compress the content, so that the source content is already minimal enough that we don't need the model to cite specific sources or spans. For example, we could break up each document into a sentence or two, embed those and keep only the most relevant ones. LangChain has some built-in components for this. Here we'll use a [RecursiveCharacterTextSplitter](https://python.langchain.com/api_reference/text_splitters/text_splitter/langchain_text_splitters.RecursiveCharacterTextSplitter.html#langchain_text_splitters.RecursiveCharacterTextSplitter), which creates chunks of a sepacified size by splitting on separator substrings, and an [EmbeddingsFilter](https://python.langchain.com/api_reference/langchain/retrievers/langchain.retrievers.document_compressors.embeddings_filter.EmbeddingsFilter.html#langchain.retrievers.document_compressors.embeddings_filter.EmbeddingsFilter), which keeps only the texts with the most relevant embeddings.\n",
    "\n",
    "This approach effectively swaps our original retriever with an updated one that compresses the documents. To start, we build the retriever:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9b14f817-4454-47b2-9eb0-2b8783a8c252",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adults weigh between 21 and 72 kg (46 and 159 lb). The cheetah is capable of running at 93 to 104 km/h (58 to 65 mph); it has evolved specialized adaptations for speed, including a light build, long thin legs and a long tail\n",
      "\n",
      "\n",
      "\n",
      "The cheetah (Acinonyx jubatus) is a large cat and the fastest land animal. It has a tawny to creamy white or pale buff fur that is marked with evenly spaced, solid black spots. The head is small and rounded, with a short snout and black tear-like facial streaks. It reaches 67–94 cm (26–37 in) at the shoulder, and the head-and-body length is between 1.1 and 1.5 m (3 ft 7 in and 4 ft 11 in)\n",
      "\n",
      "\n",
      "\n",
      "2 mph), or 171 body lengths per second. The cheetah, the fastest land mammal, scores at only 16 body lengths per second, while Anna's hummingbird has the highest known length-specific velocity attained by any vertebrate\n",
      "\n",
      "\n",
      "\n",
      "It feeds on small- to medium-sized prey, mostly weighing under 40 kg (88 lb), and prefers medium-sized ungulates such as impala, springbok and Thomson's gazelles. The cheetah typically stalks its prey within 60–100 m (200–330 ft) before charging towards it, trips it during the chase and bites its throat to suffocate it to death. It breeds throughout the year\n",
      "\n",
      "\n",
      "\n",
      "The cheetah was first described in the late 18th century. Four subspecies are recognised today that are native to Africa and central Iran. An African subspecies was introduced to India in 2022. It is now distributed mainly in small, fragmented populations in northwestern, eastern and southern Africa and central Iran\n",
      "\n",
      "\n",
      "\n",
      "The cheetah lives in three main social groups: females and their cubs, male \"coalitions\", and solitary males. While females lead a nomadic life searching for prey in large home ranges, males are more sedentary and instead establish much smaller territories in areas with plentiful prey and access to females. The cheetah is active during the day, with peaks during dawn and dusk\n",
      "\n",
      "\n",
      "\n",
      "The Southeast African cheetah (Acinonyx jubatus jubatus) is the nominate cheetah subspecies native to East and Southern Africa. The Southern African cheetah lives mainly in the lowland areas and deserts of the Kalahari, the savannahs of Okavango Delta, and the grasslands of the Transvaal region in South Africa. In Namibia, cheetahs are mostly found in farmlands\n",
      "\n",
      "\n",
      "\n",
      "Subpopulations have been called \"South African cheetah\" and \"Namibian cheetah.\"\n",
      "\n",
      "\n",
      "\n",
      "In India, four cheetahs of the subspecies are living in Kuno National Park in Madhya Pradesh after having been introduced there\n",
      "\n",
      "\n",
      "\n",
      "Acinonyx jubatus velox proposed in 1913 by Edmund Heller on basis of a cheetah that was shot by Kermit Roosevelt in June 1909 in the Kenyan highlands.\n",
      "Acinonyx rex proposed in 1927 by Reginald Innes Pocock on basis of a specimen from the Umvukwe Range in Rhodesia.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from langchain.retrievers.document_compressors import EmbeddingsFilter\n",
    "from langchain_core.runnables import RunnableParallel\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=400,\n",
    "    chunk_overlap=0,\n",
    "    separators=[\"\\n\\n\", \"\\n\", \".\", \" \"],\n",
    "    keep_separator=False,\n",
    ")\n",
    "compressor = EmbeddingsFilter(embeddings=OpenAIEmbeddings(), k=10)\n",
    "\n",
    "\n",
    "def split_and_filter(input) -> List[Document]:\n",
    "    docs = input[\"docs\"]\n",
    "    question = input[\"question\"]\n",
    "    split_docs = splitter.split_documents(docs)\n",
    "    stateful_docs = compressor.compress_documents(split_docs, question)\n",
    "    return [stateful_doc for stateful_doc in stateful_docs]\n",
    "\n",
    "\n",
    "new_retriever = (\n",
    "    RunnableParallel(question=RunnablePassthrough(), docs=retriever) | split_and_filter\n",
    ")\n",
    "docs = new_retriever.invoke(\"How fast are cheetahs?\")\n",
    "for doc in docs:\n",
    "    print(doc.page_content)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "984bc1e1-76fb-4d84-baa9-5fa5abca9da4",
   "metadata": {},
   "source": [
    "Next, we assemble it into our chain as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "fa2adb01-5d8f-484c-8216-bae35717db0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "rag_chain_from_docs = (\n",
    "    RunnablePassthrough.assign(context=(lambda x: format_docs(x[\"context\"])))\n",
    "    | prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "chain = RunnablePassthrough.assign(\n",
    "    context=(lambda x: x[\"input\"]) | new_retriever\n",
    ").assign(answer=rag_chain_from_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1a5b72f8-135b-4604-8777-59f2ef682323",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cheetahs are capable of running at speeds between 93 to 104 km/h (58 to 65 mph), making them the fastest land animals.\n"
     ]
    }
   ],
   "source": [
    "result = chain.invoke({\"input\": \"How fast are cheetahs?\"})\n",
    "\n",
    "print(result[\"answer\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ac43ab-db4f-458a-9b5a-fd3e116229bd",
   "metadata": {},
   "source": [


---
sidebar_position: 2
---

# How to install LangChain packages

The LangChain ecosystem is split into different packages, which allow you to choose exactly which pieces of
functionality to install.

## Official release

To install the main `langchain` package, run:

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from "@theme/CodeBlock";

<Tabs>
  <TabItem value="pip" label="Pip" default>
    <CodeBlock language="bash">pip install langchain</CodeBlock>
  </TabItem>
  <TabItem value="conda" label="Conda">
    <CodeBlock language="bash">conda install langchain -c conda-forge</CodeBlock>
  </TabItem>
</Tabs>

While this package acts as a sane starting point to using LangChain,
much of the value of LangChain comes when integrating it with various model providers, datastores, etc.
By default, the dependencies needed to do that are NOT installed. You will need to install the dependencies for specific integrations separately, which we show below.

## Ecosystem packages

With the exception of the `langsmith` SDK, all packages in the LangChain ecosystem depend on `langchain-core`, which contains base
classes and abstractions that other packages use. The dependency graph below shows how the difference packages are related.
A directed arrow indicates that the source package depends on the target package:

![](/img/ecosystem_packages.png)

When installing a package, you do not need to explicitly install that package's explicit dependencies (such as `langchain-core`).
However, you may choose to if you are using a feature only available in a certain version of that dependency.
If you do, you should make sure that the installed or pinned version is compatible with any other integration packages you use.

### LangChain core
The `langchain-core` package contains base abstractions that the rest of the LangChain ecosystem uses, along with the LangChain Expression Language. It is automatically installed by `langchain`, but can also be used separately. Install with:

```bash
pip install langchain-core
```

### Integration packages

Certain integrations like OpenAI and Anthropic have their own packages.
Any integrations that require their own package will be documented as such in the [Integration docs](/docs/integrations/providers/).
You can see a list of all integration packages in the [API reference](https://api.python.langchain.com) under the "Partner libs" dropdown.
To install one of these run:

```bash
pip install langchain-openai
```

Any integrations that haven't been split out into their own packages will live in the `langchain-community` package. Install with:

```bash
pip install langchain-community
```

### LangChain experimental
The `langchain-experimental` package holds experimental LangChain code, intended for research and experimental uses.
Install with:

```bash
pip install langchain-experimental
```

### LangGraph
`langgraph` is a library for building stateful, multi-actor applications with LLMs. It integrates smoothly with LangChain, but can be used without it.
Install with:

```bash
pip install langgraph
```

### LangServe
LangServe helps developers deploy LangChain runnables and chains as a REST API.
LangServe is automatically installed by LangChain CLI.
If not using LangChain CLI, install with:

```bash
pip install "langserve[all]"
```
for both client and server dependencies. Or `pip install "langserve[client]"` for client code, and `pip install "langserve[server]"` for server code.

### LangChain CLI
The LangChain CLI is useful for working with LangChain templates and other LangServe projects.
Install with:

```bash
pip install langchain-cli
```

### LangSmith SDK
The LangSmith SDK is automatically installed by LangChain. However, it does not depend on
`langchain-core`, and can be installed and used independently if desired.
If you are not using LangChain, you can install it with:

```bash
pip install langsmith
```

### From source

If you want to install a package from source, you can do so by cloning the [main LangChain repo](https://github.com/langchain-ai/langchain), enter the directory of the package you want to install `PATH/TO/REPO/langchain/libs/{package}`, and run:

```bash
pip install -e .
```

LangGraph, LangSmith SDK, and certain integration packages live outside the main LangChain repo. You can see [all repos here](https://github.com/langchain-ai).


"INFO: Processing entire page OCR with tesseract...\n",
      "INFO: Processing entire page OCR with tesseract...\n",
      "INFO: Processing entire page OCR with tesseract...\n",
      "INFO: Processing entire page OCR with tesseract...\n",
      "INFO: Processing entire page OCR with tesseract...\n",
      "INFO: Processing entire page OCR with tesseract...\n"
     ]
    }
   ],
   "source": [
    "loader_local = UnstructuredLoader(\n",
    "    file_path=file_path,\n",
    "    strategy=\"hi_res\",\n",
    ")\n",
    "docs_local = []\n",
    "for doc in loader_local.lazy_load():\n",
    "    docs_local.append(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5a7a95-c7fb-40ef-b98b-bbef2d501900",
   "metadata": {},
   "source": [
    "The list of documents can then be processed similarly to those obtained from the API.\n",
    "\n",
    "## Use of multimodal models\n",
    "\n",
    "Many modern LLMs support inference over multimodal inputs (e.g., images). In some applications-- such as question-answering over PDFs with complex layouts, diagrams, or scans-- it may be advantageous to skip the PDF parsing, instead casting a PDF page to an image and passing it to a model directly. This allows a model to reason over the two dimensional content on the page, instead of a \"one-dimensional\" string representation.\n",
    "\n",
    "In principle we can use any LangChain [chat model](/docs/concepts/#chat-models) that supports multimodal inputs. A list of these models is documented [here](/docs/integrations/chat/). Below we use OpenAI's `gpt-4o-mini`.\n",
    "\n",
    "First we define a short utility function to convert a PDF page to a base64-encoded image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9f9a7f9-bab4-4278-8d9f-dea0260a7c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU PyMuPDF pillow langchain-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ec2fbae5-f3dc-4b84-8b43-145d27c334fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import io\n",
    "\n",
    "import fitz\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "def pdf_page_to_base64(pdf_path: str, page_number: int):\n",
    "    pdf_document = fitz.open(pdf_path)\n",
    "    page = pdf_document.load_page(page_number - 1)  # input is one-indexed\n",
    "    pix = page.get_pixmap()\n",
    "    img = Image.frombytes(\"RGB\", [pix.width, pix.height], pix.samples)\n",
    "\n",
    "    buffer = io.BytesIO()\n",
    "    img.save(buffer, format=\"PNG\")\n",
    "\n",
    "    return base64.b64encode(buffer.getvalue()).decode(\"utf-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "66f157bd-6752-45f6-9f7b-b0c650045b8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Image as IPImage\n",
    "from IPython.display import display\n",
    "\n",
    "base64_image = pdf_page_to_base64(file_path, 11)\n",
    "display(IPImage(data=base64.b64decode(base64_image)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14e273e9-d35b-4701-a48e-b1d3cbe9892b",
   "metadata": {},
   "source": [
    "We can then query the model in the [usual way](/docs/how_to/multimodal_inputs/). Below we ask it a question on related to the diagram on the page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5bbad8ef-dd8b-4ab9-88c6-0a3872c658b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c5a45247-3b80-448c-979e-642741347aba",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first step in the pipeline is \"Annotate Layout Dataset.\"\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.messages import HumanMessage\n",
    "\n",
    "query = \"What is the name of the first step in the pipeline?\"\n",
    "\n",
    "message = HumanMessage(\n",
    "    content=[\n",
    "        {\"type\": \"text\", \"text\": query},\n",
    "        {\n",
    "            \"type\": \"image_url\",\n",
    "            \"image_url\": {\"url\": f\""},\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "response = llm.invoke([message])\n",
    "print(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e1853d6-4609-4eb3-a4cd-74b8f2a4e32f",
   "metadata": {},
   "source": [
    "## Other PDF loaders\n",
    "\n",
    "For a list of available LangChain PDF loaders, please see [this table](/docs/integrations/document_loaders/#pdfs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d23babb2-d538-437e-b26a-5e5e002c42a8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c95fcd15cd52c944",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "# How to split by HTML header \n",
    "## Description and motivation\n",
    "\n",
    "[HTMLHeaderTextSplitter](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html) is a \"structure-aware\" chunker that splits text at the HTML element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n",
    "\n",
    "It is analogous to the [MarkdownHeaderTextSplitter](/docs/how_to/markdown_header_metadata_splitter) for markdown files.\n",
    "\n",
    "To specify what headers to split on, specify `headers_to_split_on` when instantiating `HTMLHeaderTextSplitter` as shown below.\n",
    "\n",
    "## Usage examples\n",
    "### 1) How to split HTML strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e55d44c-1fff-449a-bf52-0d6df488323f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain-text-splitters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-02T18:57:49.208965400Z",
     "start_time": "2023-10-02T18:57:48.899756Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Foo'),\n",
       " Document(page_content='Some intro text about Foo.  \\nBar main section Bar subsection 1 Bar subsection 2', metadata={'Header 1': 'Foo'}),\n",
       " Document(page_content='Some intro text about Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section'}),\n",
       " Document(page_content='Some text about the first subtopic of Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 1'}),\n",
       " Document(page_content='Some text about the second subtopic of Bar.', metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 2'}),\n",
       " Document(page_content='Baz', metadata={'Header 1': 'Foo'}),\n",
       " Document(page_content='Some text about Baz', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}),\n",
       " Document(page_content='Some concluding text about Foo', metadata={'Header 1': 'Foo'})]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_text_splitters import HTMLHeaderTextSplitter\n",
    "\n",
    "html_string = \"\"\"\n",
    "<!DOCTYPE html>\n",
    "\n",
    "\n",
    "
\n",
    "
Foo
\n",
    "
Some intro text about Foo.
\n",
    "
\n",
    "
Bar main section
\n",
    "
Some intro text about Bar.
\n",
    "
Bar subsection 1
\n",
    "
Some text about the first subtopic of Bar.
\n",
    "
Bar subsection 2
\n",
    "
Some text about the second subtopic of Bar.
\n",
    "
\n",
    "
\n",
    "
Baz
\n",
    "
Some text about Baz
\n",
    "
\n",
    "
\n",
    "
Some concluding text about Foo
\n",
    "
\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "headers_to_split_on = [\n",
    "    (\"h1\", \"Header 1\"),\n",
    "    (\"h2\", \"Header 2\"),\n",
    "    (\"h3\", \"Header 3\"),\n",
    "]\n",
    "\n",
    "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
    "html_header_splits = html_splitter.split_text(html_string)\n",
    "html_header_splits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7126f179-f4d0-4b5d-8bef-44e83b59262c",
   "metadata": {},
   "source": [
    "To return each element together with their associated headers, specify `return_each_element=True` when instantiating `HTMLHeaderTextSplitter`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "90c23088-804c-4c89-bd09-b820587ceeef",
   "metadata": {},
   "outputs": [],
   "source": [
    "html_splitter = HTMLHeaderTextSplitter(\n",
    "    headers_to_split_on,\n",
    "    return_each_element=True,\n",
    ")\n",
    "html_header_splits_elements = html_splitter.split_text(html_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b776c54e-9159-4d88-9d6c-3a1d0b639dfe",
   "metadata": {},
   "source": [
    "Comparing with the above, where elements are aggregated by their headers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "711abc74-a7b0-4dc5-a4bb-af3cafe4e0f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Foo'\n",
      "page_content='Some intro text about Foo.  \\nBar main section Bar subsection 1 Bar subsection 2' metadata={'Header 1': 'Foo'}\n"
     ]
    }
   ],
   "source": [
    "for element in html_header_splits[:2]:\n",
    "    print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5528db-187c-418a-9480-fc0267645d42",
   "metadata": {},
   "source": [
    "Now each element is returned as a distinct `Document`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "24722d8e-d073-46a8-a821-6b722412f1be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Foo'\n",
      "page_content='Some intro text about Foo.' metadata={'Header 1': 'Foo'}\n",
      "page_content='Bar main section Bar subsection 1 Bar subsection 2' metadata={'Header 1': 'Foo'}\n"
     ]
    }
   ],
   "source": [
    "for element in html_header_splits_elements[:3]:\n",
    "    print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e29b4aade2a0070c",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "#### 2) How to split from a URL or HTML file:\n",
    "\n",
    "To read directly from a URL, pass the URL string into the `split_text_from_url` method.\n",
    "\n",


# How to recursively split text by characters

This text splitter is the recommended one for generic text. It is parameterized by a list of characters. It tries to split on them in order until the chunks are small enough. The default list is `["\n\n", "\n", " ", ""]`. This has the effect of trying to keep all paragraphs (and then sentences, and then words) together as long as possible, as those would generically seem to be the strongest semantically related pieces of text.

1. How the text is split: by list of characters.

2. How the chunk size is measured: by number of characters.

To obtain the string content directly, use `.split_text`.

To create LangChain [Document](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html) objects (e.g., for use in downstream tasks), use `.create_documents`.

%pip install -qU langchain-text-splitters

stdout

page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and'

page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.'

from langchain_text_splitters import RecursiveCharacterTextSplitter

with open("state_of_the_union.txt") as f:

# Set a really small chunk size, just to show.

['Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and',

'of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.']

Let's go through the parameters set above for `RecursiveCharacterTextSplitter`:

- `chunk_size`: The maximum size of a chunk, where size is determined by the `length_function`.

- `chunk_overlap`: Target overlap between chunks. Overlapping chunks helps to mitigate loss of information when context is divided between chunks.

- `length_function`: Function determining the chunk size.

- `is_separator_regex`: Whether the separator list (defaulting to `["\n\n", "\n", " ", ""]`) should be interpreted as regex.

## Splitting text from languages without word boundaries

Some writing systems do not have [word boundaries](https://en.wikipedia.org/wiki/Category:Writing_systems_without_word_boundaries), for example Chinese, Japanese, and Thai. Splitting text with the default separator list of `["\n\n", "\n", " ", ""]` can cause words to be split between chunks. To keep words together, you can override the list of separators to include additional punctuation:

* Add ASCII full-stop "`.`", [Unicode fullwidth](https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block)) full stop "`．`" (used in Chinese text), and [ideographic full stop](https://en.wikipedia.org/wiki/CJK_Symbols_and_Punctuation) "`。`" (used in Japanese and Chinese)

* Add [Zero-width space](https://en.wikipedia.org/wiki/Zero-width_space) used in Thai, Myanmar, Kmer, and Japanese.

* Add ASCII comma "`,`", Unicode fullwidth comma "`，`", and Unicode ideographic comma "`、`"


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "38831021-76ed-48b3-9f62-d1241a68b6ad",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 3\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a745f98b-c495-44f6-a882-757c38992d76",
   "metadata": {},
   "source": [
    "# How to use output parsers to parse an LLM response into structured format\n",
    "\n",
    "Language models output text. But there are times where you want to get more structured information than just text back. While some model providers support [built-in ways to return structured output](/docs/how_to/structured_output), not all do.\n",
    "\n",
    "Output parsers are classes that help structure language model responses. There are two main methods an output parser must implement:\n",
    "\n",
    "- \"Get format instructions\": A method which returns a string containing instructions for how the output of a language model should be formatted.\n",
    "- \"Parse\": A method which takes in a string (assumed to be the response from a language model) and parses it into some structure.\n",
    "\n",
    "And then one optional one:\n",
    "\n",
    "- \"Parse with prompt\": A method which takes in a string (assumed to be the response from a language model) and a prompt (assumed to be the prompt that generated such a response) and parses it into some structure. The prompt is largely provided in the event the OutputParser wants to retry or fix the output in some way, and needs information from the prompt to do so.\n",
    "\n",
    "## Get started\n",
    "\n",
    "Below we go over the main type of output parser, the `PydanticOutputParser`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1594b2bf-2a6f-47bb-9a81-38930f8e606b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Joke(setup='Why did the tomato turn red?', punchline='Because it saw the salad dressing!')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.output_parsers import PydanticOutputParser\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "from langchain_openai import OpenAI\n",
    "from pydantic import BaseModel, Field, model_validator\n",
    "\n",
    "model = OpenAI(model_name=\"gpt-3.5-turbo-instruct\", temperature=0.0)\n",
    "\n",
    "\n",
    "# Define your desired data structure.\n",
    "class Joke(BaseModel):\n",
    "    setup: str = Field(description=\"question to set up a joke\")\n",
    "    punchline: str = Field(description=\"answer to resolve the joke\")\n",
    "\n",
    "    # You can add custom validation logic easily with Pydantic.\n",
    "    @model_validator(mode=\"before\")\n",
    "    @classmethod\n",
    "    def question_ends_with_question_mark(cls, values: dict) -> dict:\n",
    "        setup = values[\"setup\"]\n",
    "        if setup[-1] != \"?\":\n",
    "            raise ValueError(\"Badly formed question!\")\n",
    "        return values\n",
    "\n",
    "\n",
    "# Set up a parser + inject instructions into the prompt template.\n",
    "parser = PydanticOutputParser(pydantic_object=Joke)\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
    "    input_variables=[\"query\"],\n",
    "    partial_variables={\"format_instructions\": parser.get_format_instructions()},\n",
    ")\n",
    "\n",
    "# And a query intended to prompt a language model to populate the data structure.\n",
    "prompt_and_model = prompt | model\n",
    "output = prompt_and_model.invoke({\"query\": \"Tell me a joke.\"})\n",
    "parser.invoke(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75976cd6-78e2-458b-821f-3ddf3683466b",
   "metadata": {},
   "source": [
    "## LCEL\n",
    "\n",
    "Output parsers implement the [Runnable interface](/docs/concepts#interface), the basic building block of the [LangChain Expression Language (LCEL)](/docs/concepts#langchain-expression-language-lcel). This means they support `invoke`, `ainvoke`, `stream`, `astream`, `batch`, `abatch`, `astream_log` calls.\n",
    "\n",
    "Output parsers accept a string or `BaseMessage` as input and can return an arbitrary type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "34f7ff0c-8443-4eb9-8704-b4f821811d93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Joke(setup='Why did the chicken cross the road?', punchline='To get to the other side!')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parser.invoke(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdebf4a5-57a8-4632-bd17-56723d431cf1",
   "metadata": {},
   "source": [
    "Instead of manually invoking the parser, we also could've just added it to our `Runnable` sequence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "51f7fff5-e9bd-49a1-b5ab-b9ff281b93cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Joke(setup='Why did the chicken cross the road?', punchline='To get to the other side!')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain = prompt | model | parser\n",
    "chain.invoke({\"query\": \"Tell me a joke.\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88590a0-f36b-4ad5-8a56-d300971a6440",
   "metadata": {},
   "source": [
    "While all parsers support the streaming interface, only certain parsers can stream through partially parsed objects, since this is highly dependent on the output type. Parsers which cannot construct partial objects will simply yield the fully parsed output.\n",
    "\n",
    "The `SimpleJsonOutputParser` for example can stream through partial outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d7ecfe4d-dae8-4452-98ea-e48bdc498788",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.output_parsers.json import SimpleJsonOutputParser\n",
    "\n",
    "json_prompt = PromptTemplate.from_template(\n",
    "    \"Return a JSON object with an `answer` key that answers the following question: {question}\"\n",
    ")\n",
    "json_parser = SimpleJsonOutputParser()\n",
    "json_chain = json_prompt | model | json_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "cc2b999e-47aa-41f4-ba6a-13b20a204576",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{},\n",
       " {'answer': ''},\n",
       " {'answer': 'Ant'},\n",
       " {'answer': 'Anton'},\n",
       " {'answer': 'Antonie'},\n",
       " {'answer': 'Antonie van'},\n",
       " {'answer': 'Antonie van Lee'},\n",
       " {'answer': 'Antonie van Leeu'},\n",
       " {'answer': 'Antonie van Leeuwen'},\n",


"id": "aebd704a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:54.701556Z",
     "iopub.status.busy": "2024-09-11T02:34:54.701465Z",
     "iopub.status.idle": "2024-09-11T02:34:55.179986Z",
     "shell.execute_reply": "2024-09-11T02:34:55.179640Z"
    }
   },
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "system = \"\"\"Generate a relevant search query for a library system\"\"\"\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"{question}\"),\n",
    "    ]\n",
    ")\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)\n",
    "structured_llm = llm.with_structured_output(Search)\n",
    "query_analyzer = {\"question\": RunnablePassthrough()} | prompt | structured_llm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41709a2e",
   "metadata": {},
   "source": [
    "We can see that if we spell the name exactly correctly, it knows how to handle it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cc0d344b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:55.181603Z",
     "iopub.status.busy": "2024-09-11T02:34:55.181500Z",
     "iopub.status.idle": "2024-09-11T02:34:55.778884Z",
     "shell.execute_reply": "2024-09-11T02:34:55.778324Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Search(query='aliens', author='Jesse Knight')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query_analyzer.invoke(\"what are books about aliens by Jesse Knight\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1b57eab",
   "metadata": {},
   "source": [
    "The issue is that the values you want to filter on may NOT be spelled exactly correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "82b6b2ad",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:55.784266Z",
     "iopub.status.busy": "2024-09-11T02:34:55.782603Z",
     "iopub.status.idle": "2024-09-11T02:34:56.206779Z",
     "shell.execute_reply": "2024-09-11T02:34:56.206068Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Search(query='aliens', author='Jess Knight')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query_analyzer.invoke(\"what are books about aliens by jess knight\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b60b7c2",
   "metadata": {},
   "source": [
    "### Add in all values\n",
    "\n",
    "One way around this is to add ALL possible values to the prompt. That will generally guide the query in the right direction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "98788a94",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:56.210043Z",
     "iopub.status.busy": "2024-09-11T02:34:56.209657Z",
     "iopub.status.idle": "2024-09-11T02:34:56.213962Z",
     "shell.execute_reply": "2024-09-11T02:34:56.213413Z"
    }
   },
   "outputs": [],
   "source": [
    "system = \"\"\"Generate a relevant search query for a library system.\n",
    "\n",
    "`author` attribute MUST be one of:\n",
    "\n",
    "{authors}\n",
    "\n",
    "Do NOT hallucinate author name!\"\"\"\n",
    "base_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"{question}\"),\n",
    "    ]\n",
    ")\n",
    "prompt = base_prompt.partial(authors=\", \".join(names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e65412f5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:56.216144Z",
     "iopub.status.busy": "2024-09-11T02:34:56.216005Z",
     "iopub.status.idle": "2024-09-11T02:34:56.218754Z",
     "shell.execute_reply": "2024-09-11T02:34:56.218416Z"
    }
   },
   "outputs": [],
   "source": [
    "query_analyzer_all = {\"question\": RunnablePassthrough()} | prompt | structured_llm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e639285a",
   "metadata": {},
   "source": [
    "However... if the list of categoricals is long enough, it may error!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "696b000f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:56.220827Z",
     "iopub.status.busy": "2024-09-11T02:34:56.220680Z",
     "iopub.status.idle": "2024-09-11T02:34:58.846872Z",
     "shell.execute_reply": "2024-09-11T02:34:58.846273Z"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    res = query_analyzer_all.invoke(\"what are books about aliens by jess knight\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d5d7891",
   "metadata": {},
   "source": [
    "We can try to use a longer context window... but with so much information in there, it is not garunteed to pick it up reliably"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0f0d0757",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-11T02:34:58.850318Z",
     "iopub.status.busy": "2024-09-11T02:34:58.850100Z",
     "iopub.status.idle": "2024-09-11T02:34:58.873883Z",
     "shell.execute_reply": "2024-09-11T02:34:58.873525Z"
    }
   },
   "outputs": [],
   "source": [
    "llm_long = ChatOpenAI(model=\"gpt-4-turbo-preview\", temperature=0)\n",
    "structured_llm_long = llm_long.with_structured_output(Search)\n",
    "query_analyzer_all = {\"question\": RunnablePassthrough()} | prompt | structured_llm_long"
   ]
  },
  {
   "cell_type": "code",


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bf4061ce",
   "metadata": {},
   "source": [
    "# Caching\n",
    "\n",
    "Embeddings can be stored or temporarily cached to avoid needing to recompute them.\n",
    "\n",
    "Caching embeddings can be done using a `CacheBackedEmbeddings`. The cache backed embedder is a wrapper around an embedder that caches\n",
    "embeddings in a key-value store. The text is hashed and the hash is used as the key in the cache.\n",
    "\n",
    "The main supported way to initialize a `CacheBackedEmbeddings` is `from_bytes_store`. It takes the following parameters:\n",
    "\n",
    "- underlying_embedder: The embedder to use for embedding.\n",
    "- document_embedding_cache: Any [`ByteStore`](/docs/integrations/stores/) for caching document embeddings.\n",
    "- batch_size: (optional, defaults to `None`) The number of documents to embed between store updates.\n",
    "- namespace: (optional, defaults to `\"\"`) The namespace to use for document cache. This namespace is used to avoid collisions with other caches. For example, set it to the name of the embedding model used.\n",
    "- query_embedding_cache: (optional, defaults to `None` or not caching) A [`ByteStore`](/docs/integrations/stores/) for caching query embeddings, or `True` to use the same store as `document_embedding_cache`.\n",
    "\n",
    "**Attention**:\n",
    "\n",
    "- Be sure to set the `namespace` parameter to avoid collisions of the same text embedded using different embeddings models.\n",
    "- `CacheBackedEmbeddings` does not cache query embeddings by default. To enable query caching, one need to specify a `query_embedding_cache`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a463c3c2-749b-40d1-a433-84f68a1cd1c7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.embeddings import CacheBackedEmbeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ddf07dd-3e72-41de-99d4-78e9521e272f",
   "metadata": {},
   "source": [
    "## Using with a Vector Store\n",
    "\n",
    "First, let's see an example that uses the local file system for storing embeddings and uses FAISS vector store for retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50183825",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain-openai faiss-cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9e4314d8-88ef-4f52-81ae-0be771168bb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.storage import LocalFileStore\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_community.vectorstores import FAISS\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "underlying_embeddings = OpenAIEmbeddings()\n",
    "\n",
    "store = LocalFileStore(\"./cache/\")\n",
    "\n",
    "cached_embedder = CacheBackedEmbeddings.from_bytes_store(\n",
    "    underlying_embeddings, store, namespace=underlying_embeddings.model\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8cdf33c-321d-4d2c-b76b-d6f5f8b42a92",
   "metadata": {},
   "source": [
    "The cache is empty prior to embedding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f9ad627f-ced2-4277-b336-2434f22f2c8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(store.yield_keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4effe04-b40f-42f8-a449-72fe6991cf20",
   "metadata": {},
   "source": [
    "Load the document, split it into chunks, embed each chunk and load it into the vector store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cf958ac2-e60e-4668-b32c-8bb2d78b3c61",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_documents = TextLoader(\"state_of_the_union.txt\").load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "documents = text_splitter.split_documents(raw_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f526444b-93f8-423f-b6d1-dab539450921",
   "metadata": {},
   "source": [
    "Create the vector store:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3a1d7bb8-3b72-4bb5-9013-cf7729caca61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 218 ms, sys: 29.7 ms, total: 248 ms\n",
      "Wall time: 1.02 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "db = FAISS.from_documents(documents, cached_embedder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64fc53f5-d559-467f-bf62-5daef32ffbc0",
   "metadata": {},
   "source": [
    "If we try to create the vector store again, it'll be much faster since it does not need to re-compute any embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "714cb2e2-77ba-41a8-bb83-84e75342af2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 15.7 ms, sys: 2.22 ms, total: 18 ms\n",
      "Wall time: 17.2 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "db2 = FAISS.from_documents(documents, cached_embedder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1acc76b9-9c70-4160-b593-5f932c75e2b6",
   "metadata": {},
   "source": [
    "And here are some of the embeddings that got created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f2ca32dd-3712-4093-942b-4122f3dc8a8e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['text-embedding-ada-00217a6727d-8916-54eb-b196-ec9c9d6ca472',\n",
       " 'text-embedding-ada-0025fc0d904-bd80-52da-95c9-441015bfb438',\n",
       " 'text-embedding-ada-002e4ad20ef-dfaa-5916-9459-f90c6d8e8159',\n",
       " 'text-embedding-ada-002ed199159-c1cd-5597-9757-f80498e8f17b',\n",
       " 'text-embedding-ada-0021297d37a-2bc1-5e19-bf13-6c950f075062']"
      ]
     },
     "execution_count": 8,
     "metadata": {},


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ef893cf-eac1-45e6-9eb6-72e9ca043200",
   "metadata": {},
   "source": [
    "# How to stream results from your RAG application\n",
    "\n",
    "This guide explains how to stream results from a RAG application. It covers streaming tokens from the final output as well as intermediate steps of a chain (e.g., from query re-writing).\n",
    "\n",
    "We'll work off of the Q&A app with sources we built over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng in the [RAG tutorial](/docs/tutorials/rag)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "487d8d79-5ee9-4aa4-9fdf-cd5f4303e099",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Dependencies\n",
    "\n",
    "We'll use OpenAI embeddings and a Chroma vector store in this walkthrough, but everything shown here works with any [Embeddings](/docs/concepts#embedding-models), [VectorStore](/docs/concepts#vectorstores) or [Retriever](/docs/concepts#retrievers). \n",
    "\n",
    "We'll use the following packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28d272cd-4e31-40aa-bbb4-0be0a1f49a14",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain langchain-community langchainhub langchain-openai langchain-chroma beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ef48de-70b6-4f43-8e0b-ab9b84c9c02a",
   "metadata": {},
   "source": [
    "We need to set environment variable `OPENAI_API_KEY`, which can be done directly or loaded from a `.env` file like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "143787ca-d8e6-4dc9-8281-4374f4d71720",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()\n",
    "\n",
    "# import dotenv\n",
    "\n",
    "# dotenv.load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1665e740-ce01-4f09-b9ed-516db0bd326f",
   "metadata": {},
   "source": [
    "### LangSmith\n",
    "\n",
    "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
    "\n",
    "Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07411adb-3722-4f65-ab7f-8f6f57663d11",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a72ca8-f8c8-4c0e-929a-223946c63f12",
   "metadata": {},
   "source": [
    "## RAG chain\n",
    "\n",
    "Let's first select a LLM:\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs customVarName=\"llm\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "accc4c35-e17c-4bf0-8a11-cd9e53436a3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa6ba684-26cf-4860-904e-a4d51380c134",
   "metadata": {},
   "source": [
    "Here is Q&A app with sources we built over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng in the [RAG tutorial](/docs/tutorials/rag):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "820244ae-74b4-4593-b392-822979dd91b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bs4\n",
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain_chroma import Chroma\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 1. Load, chunk and index the contents of the blog to create a retriever.\n",
    "loader = WebBaseLoader(\n",
    "    web_paths=(\"https://lilianweng.github.io/posts/2023-06-23-agent/\",),\n",
    "    bs_kwargs=dict(\n",
    "        parse_only=bs4.SoupStrainer(\n",
    "            class_=(\"post-content\", \"post-title\", \"post-header\")\n",
    "        )\n",
    "    ),\n",
    ")\n",
    "docs = loader.load()\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\n",
    "splits = text_splitter.split_documents(docs)\n",
    "vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())\n",
    "retriever = vectorstore.as_retriever()\n",
    "\n",
    "\n",
    "# 2. Incorporate the retriever into a question-answering chain.\n",
    "system_prompt = (\n",
    "    \"You are an assistant for question-answering tasks. \"\n",
    "    \"Use the following pieces of retrieved context to answer \"\n",
    "    \"the question. If you don't know the answer, say that you \"\n",
    "    \"don't know. Use three sentences maximum and keep the \"\n",
    "    \"answer concise.\"\n",
    "    \"\\n\\n\"\n",
    "    \"{context}\"\n",
    ")\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system_prompt),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "question_answer_chain = create_stuff_documents_chain(llm, prompt)\n",
    "rag_chain = create_retrieval_chain(retriever, question_answer_chain)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c2f99b5-80b4-4178-bf30-c1c0a152638f",
   "metadata": {},
   "source": [
    "## Streaming final outputs\n",
    "\n",
    "The chain constructed by `create_retrieval_chain` returns a dict with keys `\"input\"`, `\"context\"`, and `\"answer\"`. The `.stream` method will by default stream each key in a sequence.\n",
    "\n",


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "c47f5b2f-e14c-43e7-a0ab-d71562636624",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 3\n",
    "keywords: [summarize, summarization, map reduce]\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682a4f53-27db-43ef-a909-dd9ded76051b",
   "metadata": {},
   "source": [
    "# How to summarize text through parallelization\n",
    "\n",
    "LLMs can summarize and otherwise distill desired information from text, including large volumes of text. In many cases, especially when the amount of text is large compared to the size of the model's context window, it can be helpful (or necessary) to break up the summarization task into smaller components.\n",
    "\n",
    "Map-reduce represents one class of strategies for accomplishing this. The idea is to break the text into \"sub-documents\", and first map each sub-document to an individual summary using an LLM. Then, we reduce or consolidate those summaries into a single global summary.\n",
    "\n",
    "Note that the map step is typically parallelized over the input documents. This strategy is especially effective when understanding of a sub-document does not rely on preceeding context. For example, when summarizing a corpus of many, shorter documents.\n",
    "\n",
    "[LangGraph](https://langchain-ai.github.io/langgraph/), built on top of `langchain-core`, suports [map-reduce](https://langchain-ai.github.io/langgraph/how-tos/map-reduce/) workflows and is well-suited to this problem:\n",
    "\n",
    "- LangGraph allows for individual steps (such as successive summarizations) to be streamed, allowing for greater control of execution;\n",
    "- LangGraph's [checkpointing](https://langchain-ai.github.io/langgraph/how-tos/persistence/) supports error recovery, extending with human-in-the-loop workflows, and easier incorporation into conversational applications.\n",
    "- The LangGraph implementation is straightforward to modify and extend.\n",
    "\n",
    "Below, we demonstrate how to summarize text via a map-reduce strategy."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aa52e84-d1b5-4b33-b4c4-541156686ef3",
   "metadata": {},
   "source": [
    "## Load chat model\n",
    "\n",
    "Let's first load a chat model:\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs\n",
    "  customVarName=\"llm\"\n",
    "/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e5f426fc-cea6-4351-8931-1e422d3c8b69",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b137fe82-0a53-4910-b53e-b87a297f329d",
   "metadata": {},
   "source": [
    "## Load documents\n",
    "\n",
    "First we load in our documents. We will use [WebBaseLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) to load a blog post, and split the documents into smaller sub-documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "27c8fed0-b2d7-4549-a086-f5ee657efc41",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 1003, which is longer than the specified 1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated 14 documents.\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=1000, chunk_overlap=0\n",
    ")\n",
    "\n",
    "loader = WebBaseLoader(\"https://lilianweng.github.io/posts/2023-06-23-agent/\")\n",
    "docs = loader.load()\n",
    "\n",
    "split_docs = text_splitter.split_documents(docs)\n",
    "print(f\"Generated {len(split_docs)} documents.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84216044-6f1e-4b90-b4fa-29ec305abf51",
   "metadata": {},
   "source": [
    "## Create graph\n",
    "\n",
    "### Map step\n",
    "Let's first define the prompt associated with the map step, and associated it with the LLM via a [chain](/docs/how_to/sequence/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "669afa40-2708-4fa1-841e-c74a67bd9175",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "map_prompt = ChatPromptTemplate.from_messages(\n",
    "    [(\"human\", \"Write a concise summary of the following:\\\\n\\\\n{context}\")]\n",
    ")\n",
    "\n",
    "map_chain = map_prompt | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81597ed0-8df5-4cbc-a242-3140a168a7f4",
   "metadata": {},
   "source": [
    "### Reduce step\n",
    "\n",
    "We also define a chain that takes the document mapping results and reduces them into a single output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "de59caae-8fb2-4cf4-aea0-be78a081a695",
   "metadata": {},
   "outputs": [],
   "source": [
    "reduce_template = \"\"\"\n",
    "The following is a set of summaries:\n",
    "{docs}\n",
    "Take these and distill it into a final, consolidated summary\n",
    "of the main themes.\n",
    "\"\"\"\n",
    "\n",
    "reduce_prompt = ChatPromptTemplate([(\"human\", reduce_template)])\n",
    "\n",
    "reduce_chain = reduce_prompt | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb264a71-12f5-44ef-ad2e-d38c4bf71bbd",
   "metadata": {},
   "source": [
    "### Orchestration via LangGraph\n",
    "\n",
    "Below we implement a simple application that maps the summarization step on a list of documents, then reduces them using the above prompts.\n",
    "\n",
    "Map-reduce flows are particularly useful when texts are long compared to the context window of a LLM. For long texts, we need a mechanism that ensures that the context to be summarized in the reduce step does not exceed a model's context window size. Here we implement a recursive \"collapsing\" of the summaries: the inputs are partitioned based on a token limit, and summaries are generated of the partitions. This step is repeated until the total length of the summaries is within a desired limit, allowing for the summarization of arbitrary-length text.\n",
    "\n",
    "We will need to install `langgraph`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dc8cf11-c0e5-4448-a921-9377acad1df0",
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install -qU langgraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,


"id": "dafedc2e-feeb-44bc-9f38-e55394953de5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import operator\n",
    "from typing import Annotated, List, Literal, TypedDict\n",
    "\n",
    "from langchain.chains.combine_documents.reduce import (\n",
    "    acollapse_docs,\n",
    "    split_list_of_docs,\n",
    ")\n",
    "from langchain_core.documents import Document\n",
    "from langgraph.constants import Send\n",
    "from langgraph.graph import END, START, StateGraph\n",
    "\n",
    "token_max = 1000\n",
    "\n",
    "\n",
    "def length_function(documents: List[Document]) -> int:\n",
    "    \"\"\"Get number of tokens for input contents.\"\"\"\n",
    "    return sum(llm.get_num_tokens(doc.page_content) for doc in documents)\n",
    "\n",
    "\n",
    "# This will be the overall state of the main graph.\n",
    "# It will contain the input document contents, corresponding\n",
    "# summaries, and a final summary.\n",
    "class OverallState(TypedDict):\n",
    "    # Notice here we use the operator.add\n",
    "    # This is because we want combine all the summaries we generate\n",
    "    # from individual nodes back into one list - this is essentially\n",
    "    # the \"reduce\" part\n",
    "    contents: List[str]\n",
    "    summaries: Annotated[list, operator.add]\n",
    "    collapsed_summaries: List[Document]\n",
    "    final_summary: str\n",
    "\n",
    "\n",
    "# This will be the state of the node that we will \"map\" all\n",
    "# documents to in order to generate summaries\n",
    "class SummaryState(TypedDict):\n",
    "    content: str\n",
    "\n",
    "\n",
    "# Here we generate a summary, given a document\n",
    "async def generate_summary(state: SummaryState):\n",
    "    response = await map_chain.ainvoke(state[\"content\"])\n",
    "    return {\"summaries\": [response]}\n",
    "\n",
    "\n",
    "# Here we define the logic to map out over the documents\n",
    "# We will use this an edge in the graph\n",
    "def map_summaries(state: OverallState):\n",
    "    # We will return a list of `Send` objects\n",
    "    # Each `Send` object consists of the name of a node in the graph\n",
    "    # as well as the state to send to that node\n",
    "    return [\n",
    "        Send(\"generate_summary\", {\"content\": content}) for content in state[\"contents\"]\n",
    "    ]\n",
    "\n",
    "\n",
    "def collect_summaries(state: OverallState):\n",
    "    return {\n",
    "        \"collapsed_summaries\": [Document(summary) for summary in state[\"summaries\"]]\n",
    "    }\n",
    "\n",
    "\n",
    "# Add node to collapse summaries\n",
    "async def collapse_summaries(state: OverallState):\n",
    "    doc_lists = split_list_of_docs(\n",
    "        state[\"collapsed_summaries\"], length_function, token_max\n",
    "    )\n",
    "    results = []\n",
    "    for doc_list in doc_lists:\n",
    "        results.append(await acollapse_docs(doc_list, reduce_chain.ainvoke))\n",
    "\n",
    "    return {\"collapsed_summaries\": results}\n",
    "\n",
    "\n",
    "# This represents a conditional edge in the graph that determines\n",
    "# if we should collapse the summaries or not\n",
    "def should_collapse(\n",
    "    state: OverallState,\n",
    ") -> Literal[\"collapse_summaries\", \"generate_final_summary\"]:\n",
    "    num_tokens = length_function(state[\"collapsed_summaries\"])\n",
    "    if num_tokens > token_max:\n",
    "        return \"collapse_summaries\"\n",
    "    else:\n",
    "        return \"generate_final_summary\"\n",
    "\n",
    "\n",
    "# Here we will generate the final summary\n",
    "async def generate_final_summary(state: OverallState):\n",
    "    response = await reduce_chain.ainvoke(state[\"collapsed_summaries\"])\n",
    "    return {\"final_summary\": response}\n",
    "\n",
    "\n",
    "# Construct the graph\n",
    "# Nodes:\n",
    "graph = StateGraph(OverallState)\n",
    "graph.add_node(\"generate_summary\", generate_summary)  # same as before\n",
    "graph.add_node(\"collect_summaries\", collect_summaries)\n",
    "graph.add_node(\"collapse_summaries\", collapse_summaries)\n",
    "graph.add_node(\"generate_final_summary\", generate_final_summary)\n",
    "\n",
    "# Edges:\n",
    "graph.add_conditional_edges(START, map_summaries, [\"generate_summary\"])\n",
    "graph.add_edge(\"generate_summary\", \"collect_summaries\")\n",
    "graph.add_conditional_edges(\"collect_summaries\", should_collapse)\n",
    "graph.add_conditional_edges(\"collapse_summaries\", should_collapse)\n",
    "graph.add_edge(\"generate_final_summary\", END)\n",
    "\n",
    "app = graph.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2de9413-fa18-4807-9c1f-85a62a8eb7ab",
   "metadata": {},
   "source": [
    "LangGraph allows the graph structure to be plotted to help visualize its function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4f26c1e3-3d3c-44f7-bb5f-46db9dc40f4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "\n",
    "Image(app.get_graph().draw_mermaid_png())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74f3e276-f003-4112-ba14-c6952076c4f8",
   "metadata": {},
   "source": [
    "## Invoke graph\n",
    "\n",
    "When running the application, we can stream the graph to observe its sequence of steps. Below, we will simply print out the name of the step.\n",
    "\n",
    "Note that because we have a loop in the graph, it can be helpful to specify a [recursion_limit](https://langchain-ai.github.io/langgraph/reference/errors/#langgraph.errors.GraphRecursionError) on its execution. This will raise a specific error when the specified limit is exceeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0701bb7d-fbc6-497e-a577-25d56e6e43c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['generate_summary']\n",
      "['collect_summaries']\n",
      "['collapse_summaries']\n",
      "['collapse_summaries']\n",
      "['generate_final_summary']\n"
     ]
    }
   ],
   "source": [
    "async for step in app.astream(\n",
    "    {\"contents\": [doc.page_content for doc in split_docs]},\n",
    "    {\"recursion_limit\": 10},\n",
    "):\n",
    "    print(list(step.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0dc27458-7b37-4a2b-9452-b59274a55828",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",