{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5436020b",
   "metadata": {},
   "source": [
    "# How to create tools\n",
    "\n",
    "When constructing an agent, you will need to provide it with a list of `Tool`s that it can use. Besides the actual function that is called, the Tool consists of several components:\n",
    "\n",
    "| Attribute     | Type                            | Description                                                                                |\n",
    "|---------------|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
    "| name          | str                             | Must be unique within a set of tools provided to an LLM or agent.                                                                                |\n",
    "| description   | str                             | Describes what the tool does. Used as context by the LLM or agent.                                                                                |\n",
    "| args_schema   | pydantic.BaseModel | Optional but recommended, and required if using callback handlers. It can be used to provide more information (e.g., few-shot examples) or validation for expected parameters. |\n",
    "| return_direct | boolean                         | Only relevant for agents. When True, after invoking the given tool, the agent will stop and return the result direcly to the user.                                             |\n",
    "\n",
    "LangChain supports the creation of tools from:\n",
    "\n",
    "1. Functions;\n",
    "2. LangChain [Runnables](/docs/concepts#runnable-interface);\n",
    "3. By sub-classing from [BaseTool](https://python.langchain.com/api_reference/core/tools/langchain_core.tools.BaseTool.html) -- This is the most flexible method, it provides the largest degree of control, at the expense of more effort and code.\n",
    "\n",
    "Creating tools from functions may be sufficient for most use cases, and can be done via a simple [@tool decorator](https://python.langchain.com/api_reference/core/tools/langchain_core.tools.tool.html#langchain_core.tools.tool). If more configuration is needed-- e.g., specification of both sync and async implementations-- one can also use the [StructuredTool.from_function](https://python.langchain.com/api_reference/core/tools/langchain_core.tools.structured.StructuredTool.html#langchain_core.tools.structured.StructuredTool.from_function) class method.\n",
    "\n",
    "In this guide we provide an overview of these methods.\n",
    "\n",
    ":::tip\n",
    "\n",
    "Models will perform better if the tools have well chosen names, descriptions and JSON schemas.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7326b23",
   "metadata": {},
   "source": [
    "## Creating tools from functions\n",
    "\n",
    "### @tool decorator\n",
    "\n",
    "This `@tool` decorator is the simplest way to define a custom tool. The decorator uses the function name as the tool name by default, but this can be overridden by passing a string as the first argument. Additionally, the decorator will use the function's docstring as the tool's description - so a docstring MUST be provided. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cc7005cd-072f-4d37-8453-6297468e5192",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:52.645451Z",
     "iopub.status.busy": "2024-09-10T20:25:52.645081Z",
     "iopub.status.idle": "2024-09-10T20:25:53.030958Z",
     "shell.execute_reply": "2024-09-10T20:25:53.030669Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "multiply\n",
      "Multiply two numbers.\n",
      "{'a': {'title': 'A', 'type': 'integer'}, 'b': {'title': 'B', 'type': 'integer'}}\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.tools import tool\n",
    "\n",
    "\n",
    "@tool\n",
    "def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiply two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "# Let's inspect some of the attributes associated with the tool.\n",
    "print(multiply.name)\n",
    "print(multiply.description)\n",
    "print(multiply.args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96698b67-993a-4c97-b867-333132e1eb14",
   "metadata": {},
   "source": [
    "Or create an **async** implementation, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0c0991db-b997-4611-be37-4346e660506b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:53.032544Z",
     "iopub.status.busy": "2024-09-10T20:25:53.032420Z",
     "iopub.status.idle": "2024-09-10T20:25:53.035349Z",
     "shell.execute_reply": "2024-09-10T20:25:53.035123Z"
    }
   },
   "outputs": [],
   "source": [
    "from langchain_core.tools import tool\n",
    "\n",
    "\n",
    "@tool\n",
    "async def amultiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiply two numbers.\"\"\"\n",
    "    return a * b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f0edc51-c586-414c-8941-c8abe779943f",
   "metadata": {},
   "source": [
    "Note that `@tool` supports parsing of annotations, nested schemas, and other features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5626423f-053e-4a66-adca-1d794d835397",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:53.036658Z",
     "iopub.status.busy": "2024-09-10T20:25:53.036574Z",
     "iopub.status.idle": "2024-09-10T20:25:53.041154Z",
     "shell.execute_reply": "2024-09-10T20:25:53.040964Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': 'Multiply a by the maximum of b.',\n",
       " 'properties': {'a': {'description': 'scale factor',\n",
       "   'title': 'A',\n",
       "   'type': 'string'},\n",
       "  'b': {'description': 'list of ints over which to take maximum',\n",
       "   'items': {'type': 'integer'},\n",
       "   'title': 'B',\n",
       "   'type': 'array'}},\n",
       " 'required': ['a', 'b'],\n",
       " 'title': 'multiply_by_maxSchema',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import Annotated, List\n",
    "\n",
    "\n",
    "@tool\n",
    "def multiply_by_max(\n",
    "    a: Annotated[str, \"scale factor\"],\n",
    "    b: Annotated[List[int], \"list of ints over which to take maximum\"],\n",
    ") -> int:\n",
    "    \"\"\"Multiply a by the maximum of b.\"\"\"\n",
    "    return a * max(b)\n",
    "\n",
    "\n",
    "multiply_by_max.args_schema.schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d6eee9",
   "metadata": {},
   "source": [
    "You can also customize the tool name and JSON args by passing them into the tool decorator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9216d03a-f6ea-4216-b7e1-0661823a4c0b",


"\n",
    "print(multiply.invoke({\"a\": 2, \"b\": 3}))\n",
    "print(await multiply.ainvoke({\"a\": 2, \"b\": 3}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97aba6cc-4bdf-4fab-aff3-d89e7d9c3a09",
   "metadata": {},
   "source": [
    "## How to create async tools\n",
    "\n",
    "LangChain Tools implement the [Runnable interface 🏃](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.base.Runnable.html).\n",
    "\n",
    "All Runnables expose the `invoke` and `ainvoke` methods (as well as other methods like `batch`, `abatch`, `astream` etc).\n",
    "\n",
    "So even if you only provide an `sync` implementation of a tool, you could still use the `ainvoke` interface, but there\n",
    "are some important things to know:\n",
    "\n",
    "* LangChain's by default provides an async implementation that assumes that the function is expensive to compute, so it'll delegate execution to another thread.\n",
    "* If you're working in an async codebase, you should create async tools rather than sync tools, to avoid incuring a small overhead due to that thread.\n",
    "* If you need both sync and async implementations, use `StructuredTool.from_function` or sub-class from `BaseTool`.\n",
    "* If implementing both sync and async, and the sync code is fast to run, override the default LangChain async implementation and simply call the sync code.\n",
    "* You CANNOT and SHOULD NOT use the sync `invoke` with an `async` tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6615cb77-fd4c-4676-8965-f92cc71d4944",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:53.142587Z",
     "iopub.status.busy": "2024-09-10T20:25:53.142504Z",
     "iopub.status.idle": "2024-09-10T20:25:53.147205Z",
     "shell.execute_reply": "2024-09-10T20:25:53.146995Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.tools import StructuredTool\n",
    "\n",
    "\n",
    "def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiply two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "calculator = StructuredTool.from_function(func=multiply)\n",
    "\n",
    "print(calculator.invoke({\"a\": 2, \"b\": 3}))\n",
    "print(\n",
    "    await calculator.ainvoke({\"a\": 2, \"b\": 5})\n",
    ")  # Uses default LangChain async implementation incurs small overhead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bb2af583-eadd-41f4-a645-bf8748bd3dcd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:53.148383Z",
     "iopub.status.busy": "2024-09-10T20:25:53.148307Z",
     "iopub.status.idle": "2024-09-10T20:25:53.152684Z",
     "shell.execute_reply": "2024-09-10T20:25:53.152486Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.tools import StructuredTool\n",
    "\n",
    "\n",
    "def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiply two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "async def amultiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiply two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "calculator = StructuredTool.from_function(func=multiply, coroutine=amultiply)\n",
    "\n",
    "print(calculator.invoke({\"a\": 2, \"b\": 3}))\n",
    "print(\n",
    "    await calculator.ainvoke({\"a\": 2, \"b\": 5})\n",
    ")  # Uses use provided amultiply without additional overhead"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c80ffdaa-e4ba-4a70-8500-32bf4f60cc1a",
   "metadata": {},
   "source": [
    "You should not and cannot use `.invoke` when providing only an async definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4ad0932c-8610-4278-8c57-f9218f654c8a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:53.153849Z",
     "iopub.status.busy": "2024-09-10T20:25:53.153773Z",
     "iopub.status.idle": "2024-09-10T20:25:53.158312Z",
     "shell.execute_reply": "2024-09-10T20:25:53.158130Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raised not implemented error. You should not be doing this.\n"
     ]
    }
   ],
   "source": [
    "@tool\n",
    "async def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiply two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "try:\n",
    "    multiply.invoke({\"a\": 2, \"b\": 3})\n",
    "except NotImplementedError:\n",
    "    print(\"Raised not implemented error. You should not be doing this.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c746a7-88d7-4afb-bcb8-0e98b891e8b6",
   "metadata": {},
   "source": [
    "## Handling Tool Errors \n",
    "\n",
    "If you're using tools with agents, you will likely need an error handling strategy, so the agent can recover from the error and continue execution.\n",
    "\n",
    "A simple strategy is to throw a `ToolException` from inside the tool and specify an error handler using `handle_tool_error`. \n",
    "\n",
    "When the error handler is specified, the exception will be caught and the error handler will decide which output to return from the tool.\n",
    "\n",
    "You can set `handle_tool_error` to `True`, a string value, or a function. If it's a function, the function should take a `ToolException` as a parameter and return a value.\n",
    "\n",
    "Please note that only raising a `ToolException` won't be effective. You need to first set the `handle_tool_error` of the tool because its default value is `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7094c0e8-6192-4870-a942-aad5b5ae48fd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-09-10T20:25:53.159440Z",
     "iopub.status.busy": "2024-09-10T20:25:53.159364Z",
     "iopub.status.idle": "2024-09-10T20:25:53.160922Z",
     "shell.execute_reply": "2024-09-10T20:25:53.160712Z"
    }
   },


"id": "8ba1764d-0272-4f98-adcf-b48cb2c0a315",
   "metadata": {},
   "source": [
    "### Invoking the tool\n",
    "\n",
    "Great! We're able to generate tool invocations. But what if we want to actually call the tool? To do so we'll need to pass the generated tool args to our tool. As a simple example we'll just extract the arguments of the first tool_call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4f5325ca-e5dc-4d1a-ba36-b085a029c90a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "92"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from operator import itemgetter\n",
    "\n",
    "chain = llm_with_tools | (lambda x: x.tool_calls[0][\"args\"]) | multiply\n",
    "chain.invoke(\"What's four times 23\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79a9eb63-383d-4dd4-a162-08b4a52ef4d9",
   "metadata": {},
   "source": [
    "Check out the [LangSmith trace here](https://smith.langchain.com/public/16bbabb9-fc9b-41e5-a33d-487c42df4f85/r)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0521d3d5",
   "metadata": {},
   "source": [
    "## Agents\n",
    "\n",
    "Chains are great when we know the specific sequence of tool usage needed for any user input. But for certain use cases, how many times we use tools depends on the input. In these cases, we want to let the model itself decide how many times to use tools and in what order. [Agents](/docs/tutorials/agents) let us do just this.\n",
    "\n",
    "LangChain comes with a number of built-in agents that are optimized for different use cases. Read about all the [agent types here](/docs/concepts#agents).\n",
    "\n",
    "We'll use the [tool calling agent](https://python.langchain.com/api_reference/langchain/agents/langchain.agents.tool_calling_agent.base.create_tool_calling_agent.html), which is generally the most reliable kind and the recommended one for most use cases.\n",
    "\n",
    "![agent](../../static/img/tool_agent.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "21723cf4-9421-4a8d-92a6-eeeb8f4367f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "from langchain.agents import AgentExecutor, create_tool_calling_agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6be83879-9da3-4dd9-b147-a79f76affd7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================\u001b[1m System Message \u001b[0m================================\n",
      "\n",
      "You are a helpful assistant\n",
      "\n",
      "=============================\u001b[1m Messages Placeholder \u001b[0m=============================\n",
      "\n",
      "\u001b[33;1m\u001b[1;3m{chat_history}\u001b[0m\n",
      "\n",
      "================================\u001b[1m Human Message \u001b[0m=================================\n",
      "\n",
      "\u001b[33;1m\u001b[1;3m{input}\u001b[0m\n",
      "\n",
      "=============================\u001b[1m Messages Placeholder \u001b[0m=============================\n",
      "\n",
      "\u001b[33;1m\u001b[1;3m{agent_scratchpad}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Get the prompt to use - can be replaced with any prompt that includes variables \"agent_scratchpad\" and \"input\"!\n",
    "prompt = hub.pull(\"hwchase17/openai-tools-agent\")\n",
    "prompt.pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "616f9714-5b18-4eed-b88a-d38e4cb1de99",
   "metadata": {},
   "source": [
    "Agents are also great because they make it easy to use multiple tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "95c86d32-ee45-4c87-a28c-14eff19b49e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tool\n",
    "def add(first_int: int, second_int: int) -> int:\n",
    "    \"Add two integers.\"\n",
    "    return first_int + second_int\n",
    "\n",
    "\n",
    "@tool\n",
    "def exponentiate(base: int, exponent: int) -> int:\n",
    "    \"Exponentiate the base to the exponent power.\"\n",
    "    return base**exponent\n",
    "\n",
    "\n",
    "tools = [multiply, add, exponentiate]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "17b09ac6-c9b7-4340-a8a0-3d3061f7888c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct the tool calling agent\n",
    "agent = create_tool_calling_agent(llm, tools, prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "675091d2-cac9-45c4-a5d7-b760ee6c1986",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an agent executor by passing in the agent and tools\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6099ab6-2fa6-452d-b73c-7fb65daab451",
   "metadata": {},
   "source": [
    "With an agent, we can ask questions that require arbitrarily-many uses of our tools:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f7dbb240-809e-4e41-8f63-1a4636e8e26d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `exponentiate` with `{'base': 3, 'exponent': 5}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[38;5;200m\u001b[1;3m243\u001b[0m\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `add` with `{'first_int': 12, 'second_int': 3}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[33;1m\u001b[1;3m15\u001b[0m\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `multiply` with `{'first_int': 243, 'second_int': 15}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3m3645\u001b[0m\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `exponentiate` with `{'base': 405, 'exponent': 2}`\n",
      "\n",
      "\n",


"Note that we didn't need to wrap the custom function `(lambda x: x.content[:5])` in a `RunnableLambda` constructor because the `model` on the left of the pipe operator is already a Runnable. The custom function is **coerced** into a runnable. See [this section](/docs/how_to/sequence/#coercion) for more information.\n",
    "\n",
    "## Passing run metadata\n",
    "\n",
    "Runnable lambdas can optionally accept a [RunnableConfig](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.config.RunnableConfig.html#langchain_core.runnables.config.RunnableConfig) parameter, which they can use to pass callbacks, tags, and other configuration information to nested runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ff0daf0c-49dd-4d21-9772-e5fa133c5f36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'foo': 'bar'}\n",
      "Tokens Used: 62\n",
      "\tPrompt Tokens: 56\n",
      "\tCompletion Tokens: 6\n",
      "Successful Requests: 1\n",
      "Total Cost (USD): $9.6e-05\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "from langchain_core.runnables import RunnableConfig\n",
    "\n",
    "\n",
    "def parse_or_fix(text: str, config: RunnableConfig):\n",
    "    fixing_chain = (\n",
    "        ChatPromptTemplate.from_template(\n",
    "            \"Fix the following text:\\n\\n```text\\n{input}\\n```\\nError: {error}\"\n",
    "            \" Don't narrate, just respond with the fixed data.\"\n",
    "        )\n",
    "        | model\n",
    "        | StrOutputParser()\n",
    "    )\n",
    "    for _ in range(3):\n",
    "        try:\n",
    "            return json.loads(text)\n",
    "        except Exception as e:\n",
    "            text = fixing_chain.invoke({\"input\": text, \"error\": e}, config)\n",
    "    return \"Failed to parse\"\n",
    "\n",
    "\n",
    "from langchain_community.callbacks import get_openai_callback\n",
    "\n",
    "with get_openai_callback() as cb:\n",
    "    output = RunnableLambda(parse_or_fix).invoke(\n",
    "        \"{foo: bar}\", {\"tags\": [\"my-tag\"], \"callbacks\": [cb]}\n",
    "    )\n",
    "    print(output)\n",
    "    print(cb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1a5e709e-9d75-48c7-bb9c-503251990505",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'foo': 'bar'}\n",
      "Tokens Used: 62\n",
      "\tPrompt Tokens: 56\n",
      "\tCompletion Tokens: 6\n",
      "Successful Requests: 1\n",
      "Total Cost (USD): $9.6e-05\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.callbacks import get_openai_callback\n",
    "\n",
    "with get_openai_callback() as cb:\n",
    "    output = RunnableLambda(parse_or_fix).invoke(\n",
    "        \"{foo: bar}\", {\"tags\": [\"my-tag\"], \"callbacks\": [cb]}\n",
    "    )\n",
    "    print(output)\n",
    "    print(cb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "922b48bd",
   "metadata": {},
   "source": [
    "## Streaming\n",
    "\n",
    ":::note\n",
    "[RunnableLambda](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.base.RunnableLambda.html) is best suited for code that does not need to support streaming. If you need to support streaming (i.e., be able to operate on chunks of inputs and yield chunks of outputs), use [RunnableGenerator](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.base.RunnableGenerator.html) instead as in the example below.\n",
    ":::\n",
    "\n",
    "You can use generator functions (ie. functions that use the `yield` keyword, and behave like iterators) in a chain.\n",
    "\n",
    "The signature of these generators should be `Iterator[Input] -> Iterator[Output]`. Or for async generators: `AsyncIterator[Input] -> AsyncIterator[Output]`.\n",
    "\n",
    "These are useful for:\n",
    "- implementing a custom output parser\n",
    "- modifying the output of a previous step, while preserving streaming capabilities\n",
    "\n",
    "Here's an example of a custom output parser for comma-separated lists. First, we create a chain that generates such a list as text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "29f55c38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lion, tiger, wolf, gorilla, panda"
     ]
    }
   ],
   "source": [
    "from typing import Iterator, List\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(\n",
    "    \"Write a comma-separated list of 5 animals similar to: {animal}. Do not include numbers\"\n",
    ")\n",
    "\n",
    "str_chain = prompt | model | StrOutputParser()\n",
    "\n",
    "for chunk in str_chain.stream({\"animal\": \"bear\"}):\n",
    "    print(chunk, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46345323",
   "metadata": {},
   "source": [
    "Next, we define a custom function that will aggregate the currently streamed output and yield it when the model generates the next comma in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f08b8a5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['lion']\n",
      "['tiger']\n",
      "['wolf']\n",
      "['gorilla']\n",
      "['raccoon']\n"
     ]
    }
   ],
   "source": [
    "# This is a custom parser that splits an iterator of llm tokens\n",
    "# into a list of strings separated by commas\n",
    "def split_into_list(input: Iterator[str]) -> Iterator[List[str]]:\n",
    "    # hold partial input until we get a comma\n",
    "    buffer = \"\"\n",
    "    for chunk in input:\n",
    "        # add current chunk to buffer\n",
    "        buffer += chunk\n",
    "        # while there are commas in the buffer\n",
    "        while \",\" in buffer:\n",
    "            # split buffer on comma\n",
    "            comma_index = buffer.index(\",\")\n",
    "            # yield everything before the comma\n",
    "            yield [buffer[:comma_index].strip()]\n",
    "            # save the rest for the next iteration\n",
    "            buffer = buffer[comma_index + 1 :]\n",
    "    # yield the last chunk\n",
    "    yield [buffer.strip()]\n",
    "\n",
    "\n",
    "list_chain = str_chain | split_into_list\n",
    "\n",
    "for chunk in list_chain.stream({\"animal\": \"bear\"}):\n",
    "    print(chunk, flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a5adb69",
   "metadata": {},
   "source": [
    "Invoking it gives a full array of values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9ea4ddc6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['lion', 'tiger', 'wolf', 'gorilla', 'raccoon']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }


"It can be helpful to return not only tool outputs but also tool inputs. We can easily do this with LCEL by `RunnablePassthrough.assign`-ing the tool output. This will take whatever the input is to the RunnablePassrthrough components (assumed to be a dictionary) and add a key to it while still passing through everything that's currently in the input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "45404406-859d-4caa-8b9d-5838162c80a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'multiply',\n",
       " 'arguments': {'x': 13, 'y': 4.14137281},\n",
       " 'output': 53.83784653}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.runnables import RunnablePassthrough\n",
    "\n",
    "chain = (\n",
    "    prompt | model | JsonOutputParser() | RunnablePassthrough.assign(output=invoke_tool)\n",
    ")\n",
    "chain.invoke({\"input\": \"what's thirteen times 4.14137281\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1797fe82-ea35-4cba-834a-1caf9740d184",
   "metadata": {},
   "source": [
    "## What's next?\n",
    "\n",
    "This how-to guide shows the \"happy path\" when the model correctly outputs all the required tool information.\n",
    "\n",
    "In reality, if you're using more complex tools, you will start encountering errors from the model, especially for models that have not been fine tuned for tool calling and for less capable models.\n",
    "\n",
    "You will need to be prepared to add strategies to improve the output from the model; e.g.,\n",
    "\n",
    "1. Provide few shot examples.\n",
    "2. Add error handling (e.g., catch the exception and feed it back to the LLM to ask it to correct its previous output)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "beba2e0e",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 2\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb0735c0",
   "metadata": {},
   "source": [
    "# How to use few shot examples in chat models\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "- [Prompt templates](/docs/concepts/#prompt-templates)\n",
    "- [Example selectors](/docs/concepts/#example-selectors)\n",
    "- [Chat models](/docs/concepts/#chat-model)\n",
    "- [Vectorstores](/docs/concepts/#vector-stores)\n",
    "\n",
    ":::\n",
    "\n",
    "This guide covers how to prompt a chat model with example inputs and outputs. Providing the model with a few such examples is called few-shotting, and is a simple yet powerful way to guide generation and in some cases drastically improve model performance.\n",
    "\n",
    "There does not appear to be solid consensus on how best to do few-shot prompting, and the optimal prompt compilation will likely vary by model. Because of this, we provide few-shot prompt templates like the [FewShotChatMessagePromptTemplate](https://python.langchain.com/api_reference/core/prompts/langchain_core.prompts.few_shot.FewShotChatMessagePromptTemplate.html?highlight=fewshot#langchain_core.prompts.few_shot.FewShotChatMessagePromptTemplate) as a flexible starting point, and you can modify or replace them as you see fit.\n",
    "\n",
    "The goal of few-shot prompt templates are to dynamically select examples based on an input, and then format the examples in a final prompt to provide for the model.\n",
    "\n",
    "**Note:** The following code examples are for chat models only, since `FewShotChatMessagePromptTemplates` are designed to output formatted [chat messages](/docs/concepts/#message-types) rather than pure strings. For similar few-shot prompt examples for pure string templates compatible with completion models (LLMs), see the [few-shot prompt templates](/docs/how_to/few_shot_examples/) guide."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d716f2de-cc29-4823-9360-a808c7bfdb86",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Fixed Examples\n",
    "\n",
    "The most basic (and common) few-shot prompting technique is to use fixed prompt examples. This way you can select a chain, evaluate it, and avoid worrying about additional moving parts in production.\n",
    "\n",
    "The basic components of the template are:\n",
    "- `examples`: A list of dictionary examples to include in the final prompt.\n",
    "- `example_prompt`: converts each example into 1 or more messages through its [`format_messages`](https://python.langchain.com/api_reference/core/prompts/langchain_core.prompts.chat.ChatPromptTemplate.html?highlight=format_messages#langchain_core.prompts.chat.ChatPromptTemplate.format_messages) method. A common example would be to convert each example into one human message and one AI message response, or a human message followed by a function call message.\n",
    "\n",
    "Below is a simple demonstration. First, define the examples you'd like to include. Let's give the LLM an unfamiliar mathematical operator, denoted by the \"🦜\" emoji:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5b79e400",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain langchain-openai langchain-chroma\n",
    "\n",
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30856d92",
   "metadata": {},
   "source": [
    "If we try to ask the model what the result of this expression is, it will fail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "174dec5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='The expression \"2 🦜 9\" is not a standard mathematical operation or equation. It appears to be a combination of the number 2 and the parrot emoji 🦜 followed by the number 9. It does not have a specific mathematical meaning.', response_metadata={'token_usage': {'completion_tokens': 54, 'prompt_tokens': 17, 'total_tokens': 71}, 'model_name': 'gpt-4o-mini', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-aad12dda-5c47-4a1e-9949-6fe94e03242a-0', usage_metadata={'input_tokens': 17, 'output_tokens': 54, 'total_tokens': 71})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "model = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0.0)\n",
    "\n",
    "model.invoke(\"What is 2 🦜 9?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6d58385",
   "metadata": {},
   "source": [
    "Now let's see what happens if we give the LLM some examples to work with. We'll define some below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0fc5a02a-6249-4e92-95c3-30fff9671e8b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate\n",
    "\n",
    "examples = [\n",
    "    {\"input\": \"2 🦜 2\", \"output\": \"4\"},\n",
    "    {\"input\": \"2 🦜 3\", \"output\": \"5\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8710ecc-2aa0-4172-a74c-250f6bc3d9e2",
   "metadata": {},
   "source": [
    "Next, assemble them into the few-shot prompt template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "65e72ad1-9060-47d0-91a1-bc130c8b98ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[HumanMessage(content='2 🦜 2'), AIMessage(content='4'), HumanMessage(content='2 🦜 3'), AIMessage(content='5')]\n"
     ]
    }
   ],
   "source": [
    "# This is a prompt template used to format each individual example.\n",
    "example_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"human\", \"{input}\"),\n",
    "        (\"ai\", \"{output}\"),\n",
    "    ]\n",
    ")\n",
    "few_shot_prompt = FewShotChatMessagePromptTemplate(\n",
    "    example_prompt=example_prompt,\n",
    "    examples=examples,\n",
    ")\n",
    "\n",
    "print(few_shot_prompt.invoke({}).to_messages())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5490bd59-b28f-46a4-bbdf-0191802dd3c5",
   "metadata": {},
   "source": [
    "Finally, we assemble the final prompt as shown below, passing `few_shot_prompt` directly into the `from_messages` factory method, and use it with a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9f86d6d9-50de-41b6-b6c7-0f9980cc0187",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [


# How to combine results from multiple retrievers

The [EnsembleRetriever](https://python.langchain.com/api_reference/langchain/retrievers/langchain.retrievers.ensemble.EnsembleRetriever.html) supports ensembling of results from multiple retrievers. It is initialized with a list of [BaseRetriever](https://python.langchain.com/api_reference/core/retrievers/langchain_core.retrievers.BaseRetriever.html) objects. EnsembleRetrievers rerank the results of the constituent retrievers based on the [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) algorithm.

By leveraging the strengths of different algorithms, the `EnsembleRetriever` can achieve better performance than any single algorithm.

The most common pattern is to combine a sparse retriever (like BM25) with a dense retriever (like embedding similarity), because their strengths are complementary. It is also known as "hybrid search". The sparse retriever is good at finding relevant documents based on keywords, while the dense retriever is good at finding relevant documents based on semantic similarity.

Below we demonstrate ensembling of a [BM25Retriever](https://python.langchain.com/api_reference/community/retrievers/langchain_community.retrievers.bm25.BM25Retriever.html) with a retriever derived from the [FAISS vector store](https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.faiss.FAISS.html).

%pip install --upgrade --quiet  rank_bm25 > /dev/null

from langchain.retrievers import EnsembleRetriever

from langchain_community.retrievers import BM25Retriever

from langchain_community.vectorstores import FAISS

from langchain_openai import OpenAIEmbeddings

# initialize the bm25 retriever and faiss retriever

doc_list_1, metadatas=[{"source": 1}] * len(doc_list_1)

doc_list_2, embedding, metadatas=[{"source": 2}] * len(doc_list_2)

faiss_retriever = faiss_vectorstore.as_retriever(search_kwargs={"k": 2})

retrievers=[bm25_retriever, faiss_retriever], weights=[0.5, 0.5]

[Document(page_content='I like apples', metadata={'source': 1}),

Document(page_content='You like apples', metadata={'source': 2}),

Document(page_content='Apples and oranges are fruits', metadata={'source': 1}),

Document(page_content='You like oranges', metadata={'source': 2})]

We can also configure the individual retrievers at runtime using [configurable fields](/docs/how_to/configure). Below we update the "top-k" parameter for the FAISS retriever specifically:

from langchain_core.runnables import ConfigurableField

[Document(page_content='I like apples', metadata={'source': 1}),

Document(page_content='You like apples', metadata={'source': 2}),

Document(page_content='Apples and oranges are fruits', metadata={'source': 1})]

config = {"configurable": {"search_kwargs_faiss": {"k": 1}}}

docs = ensemble_retriever.invoke("apples", config=config)

Notice that this only returns one source from the FAISS retriever, because we pass in the relevant configuration at run time


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "72b1b316",
   "metadata": {},
   "source": [
    "# How to parse JSON output\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "- [Chat models](/docs/concepts/#chat-models)\n",
    "- [Output parsers](/docs/concepts/#output-parsers)\n",
    "- [Prompt templates](/docs/concepts/#prompt-templates)\n",
    "- [Structured output](/docs/how_to/structured_output)\n",
    "- [Chaining runnables together](/docs/how_to/sequence/)\n",
    "\n",
    ":::\n",
    "\n",
    "While some model providers support [built-in ways to return structured output](/docs/how_to/structured_output), not all do. We can use an output parser to help users to specify an arbitrary JSON schema via the prompt, query a model for outputs that conform to that schema, and finally parse that schema as JSON.\n",
    "\n",
    ":::note\n",
    "Keep in mind that large language models are leaky abstractions! You'll have to use an LLM with sufficient capacity to generate well-formed JSON.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae909b7a",
   "metadata": {},
   "source": [
    "The [`JsonOutputParser`](https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.json.JsonOutputParser.html) is one built-in option for prompting for and then parsing JSON output. While it is similar in functionality to the [`PydanticOutputParser`](https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.pydantic.PydanticOutputParser.html), it also supports streaming back partial JSON objects.\n",
    "\n",
    "Here's an example of how it can be used alongside [Pydantic](https://docs.pydantic.dev/) to conveniently declare the expected schema:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd9d9110",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain langchain-openai\n",
    "\n",
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4ccf45a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'setup': \"Why couldn't the bicycle stand up by itself?\",\n",
       " 'punchline': 'Because it was two tired!'}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.output_parsers import JsonOutputParser\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "from langchain_openai import ChatOpenAI\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "model = ChatOpenAI(temperature=0)\n",
    "\n",
    "\n",
    "# Define your desired data structure.\n",
    "class Joke(BaseModel):\n",
    "    setup: str = Field(description=\"question to set up a joke\")\n",
    "    punchline: str = Field(description=\"answer to resolve the joke\")\n",
    "\n",
    "\n",
    "# And a query intented to prompt a language model to populate the data structure.\n",
    "joke_query = \"Tell me a joke.\"\n",
    "\n",
    "# Set up a parser + inject instructions into the prompt template.\n",
    "parser = JsonOutputParser(pydantic_object=Joke)\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
    "    input_variables=[\"query\"],\n",
    "    partial_variables={\"format_instructions\": parser.get_format_instructions()},\n",
    ")\n",
    "\n",
    "chain = prompt | model | parser\n",
    "\n",
    "chain.invoke({\"query\": joke_query})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ffa2e3",
   "metadata": {},
   "source": [
    "Note that we are passing `format_instructions` from the parser directly into the prompt. You can and should experiment with adding your own formatting hints in the other parts of your prompt to either augment or replace the default instructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "72de9c82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The output should be formatted as a JSON instance that conforms to the JSON schema below.\\n\\nAs an example, for the schema {\"properties\": {\"foo\": {\"title\": \"Foo\", \"description\": \"a list of strings\", \"type\": \"array\", \"items\": {\"type\": \"string\"}}}, \"required\": [\"foo\"]}\\nthe object {\"foo\": [\"bar\", \"baz\"]} is a well-formatted instance of the schema. The object {\"properties\": {\"foo\": [\"bar\", \"baz\"]}} is not well-formatted.\\n\\nHere is the output schema:\\n```\\n{\"properties\": {\"setup\": {\"title\": \"Setup\", \"description\": \"question to set up a joke\", \"type\": \"string\"}, \"punchline\": {\"title\": \"Punchline\", \"description\": \"answer to resolve the joke\", \"type\": \"string\"}}, \"required\": [\"setup\", \"punchline\"]}\\n```'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parser.get_format_instructions()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d801be",
   "metadata": {},
   "source": [
    "## Streaming\n",
    "\n",
    "As mentioned above, a key difference between the `JsonOutputParser` and the `PydanticOutputParser` is that the `JsonOutputParser` output parser supports streaming partial chunks. Here's what that looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0309256d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n",
      "{'setup': ''}\n",
      "{'setup': 'Why'}\n",
      "{'setup': 'Why couldn'}\n",
      "{'setup': \"Why couldn't\"}\n",
      "{'setup': \"Why couldn't the\"}\n",
      "{'setup': \"Why couldn't the bicycle\"}\n",
      "{'setup': \"Why couldn't the bicycle stand\"}\n",
      "{'setup': \"Why couldn't the bicycle stand up\"}\n",
      "{'setup': \"Why couldn't the bicycle stand up by\"}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself\"}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\"}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': ''}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': 'Because'}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': 'Because it'}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': 'Because it was'}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': 'Because it was two'}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': 'Because it was two tired'}\n",
      "{'setup': \"Why couldn't the bicycle stand up by itself?\", 'punchline': 'Because it was two tired!'}\n"
     ]
    }
   ],
   "source": [
    "for s in chain.stream({\"query\": joke_query}):\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344bd968",
   "metadata": {},
   "source": [


# How to create and query vector stores

:::info
Head to [Integrations](/docs/integrations/vectorstores/) for documentation on built-in integrations with 3rd-party vector stores.
:::

One of the most common ways to store and search over unstructured data is to embed it and store the resulting embedding
vectors, and then at query time to embed the unstructured query and retrieve the embedding vectors that are
'most similar' to the embedded query. A vector store takes care of storing embedded data and performing vector search
for you.

## Get started

This guide showcases basic functionality related to vector stores. A key part of working with vector stores is creating the vector to put in them,
which is usually created via embeddings. Therefore, it is recommended that you familiarize yourself with the [text embedding model interfaces](/docs/how_to/embed_text) before diving into this.

Before using the vectorstore at all, we need to load some data and initialize an embedding model.

We want to use OpenAIEmbeddings so we have to get the OpenAI API Key.

```python
import os
import getpass

os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')
```

```python
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter

# Load the document, split it into chunks, embed each chunk and load it into the vector store.
raw_documents = TextLoader('state_of_the_union.txt').load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
documents = text_splitter.split_documents(raw_documents)
```

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

There are many great vector store options, here are a few that are free, open-source, and run entirely on your local machine. Review all integrations for many great hosted offerings.

<Tabs>
  <TabItem value="chroma" label="Chroma" default>

This walkthrough uses the `chroma` vector database, which runs on your local machine as a library.

```bash
pip install langchain-chroma
```

```python
from langchain_chroma import Chroma

db = Chroma.from_documents(documents, OpenAIEmbeddings())
```

  </TabItem>
  <TabItem value="faiss" label="FAISS">

This walkthrough uses the `FAISS` vector database, which makes use of the Facebook AI Similarity Search (FAISS) library.

```bash
pip install faiss-cpu
```

```python
from langchain_community.vectorstores import FAISS

db = FAISS.from_documents(documents, OpenAIEmbeddings())
```

  </TabItem>
  <TabItem value="lance" label="Lance">

This notebook shows how to use functionality related to the LanceDB vector database based on the Lance data format.

```bash
pip install lancedb
```

```python
from langchain_community.vectorstores import LanceDB

import lancedb

db = lancedb.connect("/tmp/lancedb")
table = db.create_table(
    "my_table",
    data=[
        {
            "vector": embeddings.embed_query("Hello World"),
            "text": "Hello World",
            "id": "1",
        }
    ],
    mode="overwrite",
)
db = LanceDB.from_documents(documents, OpenAIEmbeddings())
```

  </TabItem>
</Tabs>

## Similarity search

All vectorstores expose a `similarity_search` method.
This will take incoming documents, create an embedding of them, and then find all documents with the most similar embedding.

```python
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
print(docs[0].page_content)
```

```output
    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

    Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
```

### Similarity search by vector

It is also possible to do a search for documents similar to a given embedding vector using `similarity_search_by_vector` which accepts an embedding vector as a parameter instead of a string.

```python
embedding_vector = OpenAIEmbeddings().embed_query(query)
docs = db.similarity_search_by_vector(embedding_vector)
print(docs[0].page_content)
```

```output
    Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.

    Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.

    One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.

    And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
```

## Async Operations


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "27598444",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "sidebar_position: 3\n",
    "keywords: [structured output, json, information extraction, with_structured_output]\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e3f0f72",
   "metadata": {},
   "source": [
    "# How to return structured data from a model\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "- [Chat models](/docs/concepts/#chat-models)\n",
    "- [Function/tool calling](/docs/concepts/#functiontool-calling)\n",
    ":::\n",
    "\n",
    "It is often useful to have a model return output that matches a specific schema. One common use-case is extracting data from text to insert into a database or use with some other downstream system. This guide covers a few strategies for getting structured outputs from a model.\n",
    "\n",
    "## The `.with_structured_output()` method\n",
    "\n",
    "\n",
    "\n",
    ":::info Supported models\n",
    "\n",
    "You can find a [list of models that support this method here](/docs/integrations/chat/).\n",
    "\n",
    ":::\n",
    "\n",
    "This is the easiest and most reliable way to get structured outputs. `with_structured_output()` is implemented for models that provide native APIs for structuring outputs, like tool/function calling or JSON mode, and makes use of these capabilities under the hood.\n",
    "\n",
    "This method takes a schema as input which specifies the names, types, and descriptions of the desired output attributes. The method returns a model-like Runnable, except that instead of outputting strings or Messages it outputs objects corresponding to the given schema. The schema can be specified as a TypedDict class, [JSON Schema](https://json-schema.org/) or a Pydantic class. If TypedDict or JSON Schema are used then a dictionary will be returned by the Runnable, and if a Pydantic class is used then a Pydantic object will be returned.\n",
    "\n",
    "As an example, let's get a model to generate a joke and separate the setup from the punchline:\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs\n",
    "  customVarName=\"llm\"\n",
    "/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6d55008f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-4o\", temperature=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a808a401-be1f-49f9-ad13-58dd68f7db5f",
   "metadata": {},
   "source": [
    "### Pydantic class\n",
    "\n",
    "If we want the model to return a Pydantic object, we just need to pass in the desired Pydantic class. The key advantage of using Pydantic is that the model-generated output will be validated. Pydantic will raise an error if any required fields are missing or if any fields are of the wrong type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "070bf702",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Joke(setup='Why was the cat sitting on the computer?', punchline='Because it wanted to keep an eye on the mouse!', rating=7)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import Optional\n",
    "\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "\n",
    "# Pydantic\n",
    "class Joke(BaseModel):\n",
    "    \"\"\"Joke to tell user.\"\"\"\n",
    "\n",
    "    setup: str = Field(description=\"The setup of the joke\")\n",
    "    punchline: str = Field(description=\"The punchline to the joke\")\n",
    "    rating: Optional[int] = Field(\n",
    "        default=None, description=\"How funny the joke is, from 1 to 10\"\n",
    "    )\n",
    "\n",
    "\n",
    "structured_llm = llm.with_structured_output(Joke)\n",
    "\n",
    "structured_llm.invoke(\"Tell me a joke about cats\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00890a47-3cdf-4805-b8f1-6d110f0633d3",
   "metadata": {},
   "source": [
    ":::tip\n",
    "Beyond just the structure of the Pydantic class, the name of the Pydantic class, the docstring, and the names and provided descriptions of parameters are very important. Most of the time `with_structured_output` is using a model's function/tool calling API, and you can effectively think of all of this information as being added to the model prompt.\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deddb6d3",
   "metadata": {},
   "source": [
    "### TypedDict or JSON Schema\n",
    "\n",
    "If you don't want to use Pydantic, explicitly don't want validation of the arguments, or want to be able to stream the model outputs, you can define your schema using a TypedDict class. We can optionally use a special `Annotated` syntax supported by LangChain that allows you to specify the default value and description of a field. Note, the default value is *not* filled in automatically if the model doesn't generate it, it is only used in defining the schema that is passed to the model.\n",
    "\n",
    ":::info Requirements\n",
    "\n",
    "- Core: `langchain-core>=0.2.26`\n",
    "- Typing extensions: It is highly recommended to import `Annotated` and `TypedDict` from `typing_extensions` instead of `typing` to ensure consistent behavior across Python versions.\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "70d82891-42e8-424a-919e-07d83bcfec61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'setup': 'Why was the cat sitting on the computer?',\n",
       " 'punchline': 'Because it wanted to keep an eye on the mouse!',\n",
       " 'rating': 7}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing_extensions import Annotated, TypedDict\n",
    "\n",
    "\n",
    "# TypedDict\n",
    "class Joke(TypedDict):\n",
    "    \"\"\"Joke to tell user.\"\"\"\n",
    "\n",
    "    setup: Annotated[str, ..., \"The setup of the joke\"]\n",
    "\n",
    "    # Alternatively, we could have specified setup as:\n",
    "\n",
    "    # setup: str                    # no default, no description\n",
    "    # setup: Annotated[str, ...]    # no default, no description\n",
    "    # setup: Annotated[str, \"foo\"]  # default, no description\n",
    "\n",
    "    punchline: Annotated[str, ..., \"The punchline of the joke\"]\n",
    "    rating: Annotated[Optional[int], None, \"How funny the joke is, from 1 to 10\"]\n",
    "\n",
    "\n",
    "structured_llm = llm.with_structured_output(Joke)\n",
    "\n",
    "structured_llm.invoke(\"Tell me a joke about cats\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4d7b4dc-f617-4ea8-aa58-847c228791b4",
   "metadata": {},
   "source": [


"For models that support more than one means of structuring outputs (i.e., they support both tool calling and JSON mode), you can specify which method to use with the `method=` argument.\n",
    "\n",
    ":::info JSON mode\n",
    "\n",
    "If using JSON mode you'll have to still specify the desired schema in the model prompt. The schema you pass to `with_structured_output` will only be used for parsing the model outputs, it will not be passed to the model the way it is with tool calling.\n",
    "\n",
    "To see if the model you're using supports JSON mode, check its entry in the [API reference](https://python.langchain.com/api_reference/langchain/index.html).\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "df0370e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'setup': 'Why was the cat sitting on the computer?',\n",
       " 'punchline': 'Because it wanted to keep an eye on the mouse!'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "structured_llm = llm.with_structured_output(None, method=\"json_mode\")\n",
    "\n",
    "structured_llm.invoke(\n",
    "    \"Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e95aa2",
   "metadata": {},
   "source": [
    "### (Advanced) Raw outputs\n",
    "\n",
    "LLMs aren't perfect at generating structured output, especially as schemas become complex. You can avoid raising exceptions and handle the raw output yourself by passing `include_raw=True`. This changes the output format to contain the raw message output, the `parsed` value (if successful), and any resulting errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "10ed2842",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'raw': AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_f25ZRmh8u5vHlOWfTUw8sJFZ', 'function': {'arguments': '{\"setup\":\"Why was the cat sitting on the computer?\",\"punchline\":\"Because it wanted to keep an eye on the mouse!\",\"rating\":7}', 'name': 'Joke'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 33, 'prompt_tokens': 93, 'total_tokens': 126}, 'model_name': 'gpt-4o-2024-05-13', 'system_fingerprint': 'fp_4e2b2da518', 'finish_reason': 'stop', 'logprobs': None}, id='run-d880d7e2-df08-4e9e-ad92-dfc29f2fd52f-0', tool_calls=[{'name': 'Joke', 'args': {'setup': 'Why was the cat sitting on the computer?', 'punchline': 'Because it wanted to keep an eye on the mouse!', 'rating': 7}, 'id': 'call_f25ZRmh8u5vHlOWfTUw8sJFZ', 'type': 'tool_call'}], usage_metadata={'input_tokens': 93, 'output_tokens': 33, 'total_tokens': 126}),\n",
       " 'parsed': {'setup': 'Why was the cat sitting on the computer?',\n",
       "  'punchline': 'Because it wanted to keep an eye on the mouse!',\n",
       "  'rating': 7},\n",
       " 'parsing_error': None}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "structured_llm = llm.with_structured_output(Joke, include_raw=True)\n",
    "\n",
    "structured_llm.invoke(\"Tell me a joke about cats\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e92a98a",
   "metadata": {},
   "source": [
    "## Prompting and parsing model outputs directly\n",
    "\n",
    "Not all models support `.with_structured_output()`, since not all models have tool calling or JSON mode support. For such models you'll need to directly prompt the model to use a specific format, and use an output parser to extract the structured response from the raw model output.\n",
    "\n",
    "### Using `PydanticOutputParser`\n",
    "\n",
    "The following example uses the built-in [`PydanticOutputParser`](https://python.langchain.com/api_reference/core/output_parsers/langchain_core.output_parsers.pydantic.PydanticOutputParser.html) to parse the output of a chat model prompted to match the given Pydantic schema. Note that we are adding `format_instructions` directly to the prompt from a method on the parser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6e514455",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from langchain_core.output_parsers import PydanticOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "\n",
    "class Person(BaseModel):\n",
    "    \"\"\"Information about a person.\"\"\"\n",
    "\n",
    "    name: str = Field(..., description=\"The name of the person\")\n",
    "    height_in_meters: float = Field(\n",
    "        ..., description=\"The height of the person expressed in meters.\"\n",
    "    )\n",
    "\n",
    "\n",
    "class People(BaseModel):\n",
    "    \"\"\"Identifying information about all people in a text.\"\"\"\n",
    "\n",
    "    people: List[Person]\n",
    "\n",
    "\n",
    "# Set up a parser\n",
    "parser = PydanticOutputParser(pydantic_object=People)\n",
    "\n",
    "# Prompt\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\n",
    "            \"system\",\n",
    "            \"Answer the user query. Wrap the output in `json` tags\\n{format_instructions}\",\n",
    "        ),\n",
    "        (\"human\", \"{query}\"),\n",
    "    ]\n",
    ").partial(format_instructions=parser.get_format_instructions())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "082fa166",
   "metadata": {},
   "source": [
    "Let’s take a look at what information is sent to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3d73d33d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System: Answer the user query. Wrap the output in `json` tags\n",
      "The output should be formatted as a JSON instance that conforms to the JSON schema below.\n",
      "\n",
      "As an example, for the schema {\"properties\": {\"foo\": {\"title\": \"Foo\", \"description\": \"a list of strings\", \"type\": \"array\", \"items\": {\"type\": \"string\"}}}, \"required\": [\"foo\"]}\n",
      "the object {\"foo\": [\"bar\", \"baz\"]} is a well-formatted instance of the schema. The object {\"properties\": {\"foo\": [\"bar\", \"baz\"]}} is not well-formatted.\n",
      "\n",
      "Here is the output schema:\n",
      "```\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4da7ae91-4973-4e97-a570-fa24024ec65d",
   "metadata": {},
   "source": [
    "# How to do query validation as part of SQL question-answering\n",
    "\n",
    "Perhaps the most error-prone part of any SQL chain or agent is writing valid and safe SQL queries. In this guide we'll go over some strategies for validating our queries and handling invalid queries.\n",
    "\n",
    "We will cover: \n",
    "\n",
    "1. Appending a \"query validator\" step to the query generation;\n",
    "2. Prompt engineering to reduce the incidence of errors.\n",
    "\n",
    "## Setup\n",
    "\n",
    "First, get required packages and set environment variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d40d5bc-3647-4b5d-808a-db470d40fe7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain langchain-community langchain-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71f46270-e1c6-45b4-b36e-ea2e9f860eba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the below to use LangSmith. Not required.\n",
    "# import os\n",
    "# os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0a2151b-cecf-4559-92a1-ca48824fed18",
   "metadata": {},
   "source": [
    "The below example will use a SQLite connection with Chinook database. Follow [these installation steps](https://database.guide/2-sample-databases-sqlite/) to create `Chinook.db` in the same directory as this notebook:\n",
    "\n",
    "* Save [this file](https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite.sql) as `Chinook_Sqlite.sql`\n",
    "* Run `sqlite3 Chinook.db`\n",
    "* Run `.read Chinook_Sqlite.sql`\n",
    "* Test `SELECT * FROM Artist LIMIT 10;`\n",
    "\n",
    "Now, `Chinhook.db` is in our directory and we can interface with it using the SQLAlchemy-driven `SQLDatabase` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8cedc936-5268-4bfa-b838-bdcc1ee9573c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sqlite\n",
      "['Album', 'Artist', 'Customer', 'Employee', 'Genre', 'Invoice', 'InvoiceLine', 'MediaType', 'Playlist', 'PlaylistTrack', 'Track']\n",
      "[(1, 'AC/DC'), (2, 'Accept'), (3, 'Aerosmith'), (4, 'Alanis Morissette'), (5, 'Alice In Chains'), (6, 'Antônio Carlos Jobim'), (7, 'Apocalyptica'), (8, 'Audioslave'), (9, 'BackBeat'), (10, 'Billy Cobham')]\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.utilities import SQLDatabase\n",
    "\n",
    "db = SQLDatabase.from_uri(\"sqlite:///Chinook.db\")\n",
    "print(db.dialect)\n",
    "print(db.get_usable_table_names())\n",
    "print(db.run(\"SELECT * FROM Artist LIMIT 10;\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d203315-fab7-4621-80da-41e9bf82d803",
   "metadata": {},
   "source": [
    "## Query checker\n",
    "\n",
    "Perhaps the simplest strategy is to ask the model itself to check the original query for common mistakes. Suppose we have the following SQL query chain:\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs customVarName=\"llm\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d81ebf69-75ad-4c92-baa9-fd152b8e622a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ec66bb76-b1ad-48ad-a7d4-b518e9421b86",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_sql_query_chain\n",
    "\n",
    "chain = create_sql_query_chain(llm, db)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da01023d-cc05-43e3-a38d-ed9d56d3ad15",
   "metadata": {},
   "source": [
    "And we want to validate its outputs. We can do so by extending the chain with a second prompt and model call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "16686750-d8ee-4c60-8d67-b28281cb6164",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "system = \"\"\"Double check the user's {dialect} query for common mistakes, including:\n",
    "- Using NOT IN with NULL values\n",
    "- Using UNION when UNION ALL should have been used\n",
    "- Using BETWEEN for exclusive ranges\n",
    "- Data type mismatch in predicates\n",
    "- Properly quoting identifiers\n",
    "- Using the correct number of arguments for functions\n",
    "- Casting to the correct data type\n",
    "- Using the proper columns for joins\n",
    "\n",
    "If there are any of the above mistakes, rewrite the query.\n",
    "If there are no mistakes, just reproduce the original query with no further commentary.\n",
    "\n",
    "Output the final SQL query only.\"\"\"\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [(\"system\", system), (\"human\", \"{query}\")]\n",
    ").partial(dialect=db.dialect)\n",
    "validation_chain = prompt | llm | StrOutputParser()\n",
    "\n",
    "full_chain = {\"query\": chain} | validation_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "28ef9c6e-21fa-4b62-8aa4-8cd398ce4c4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SELECT AVG(i.Total) AS AverageInvoice\n",
      "FROM Invoice i\n",
      "JOIN Customer c ON i.CustomerId = c.CustomerId\n",
      "WHERE c.Country = 'USA'\n",
      "AND c.Fax IS NULL\n",
      "AND i.InvoiceDate >= '2003-01-01' \n",
      "AND i.InvoiceDate < '2010-01-01'\n"
     ]
    }
   ],
   "source": [
    "query = full_chain.invoke(\n",
    "    {\n",
    "        \"question\": \"What's the average Invoice from an American customer whose Fax is missing since 2003 but before 2010\"\n",
    "    }\n",
    ")\n",
    "print(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228a1b87-4e44-4d86-bed7-fd2d7a91fb23",
   "metadata": {},
   "source": [


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "b5fc1fc7-c4c5-418f-99da-006c604a7ea6",
   "metadata": {},
   "source": [
    "---\n",
    "title: Custom Retriever\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff6f3c79-0848-4956-9115-54f6b2134587",
   "metadata": {},
   "source": [
    "# How to create a custom Retriever\n",
    "\n",
    "## Overview\n",
    "\n",
    "Many LLM applications involve retrieving information from external data sources using a `Retriever`. \n",
    "\n",
    "A retriever is responsible for retrieving a list of relevant `Documents` to a given user `query`.\n",
    "\n",
    "The retrieved documents are often formatted into prompts that are fed into an LLM, allowing the LLM to use the information in the to generate an appropriate response (e.g., answering a user question based on a knowledge base).\n",
    "\n",
    "## Interface\n",
    "\n",
    "To create your own retriever, you need to extend the `BaseRetriever` class and implement the following methods:\n",
    "\n",
    "| Method                         | Description                                      | Required/Optional |\n",
    "|--------------------------------|--------------------------------------------------|-------------------|\n",
    "| `_get_relevant_documents`      | Get documents relevant to a query.               | Required          |\n",
    "| `_aget_relevant_documents`     | Implement to provide async native support.       | Optional          |\n",
    "\n",
    "\n",
    "The logic inside of `_get_relevant_documents` can involve arbitrary calls to a database or to the web using requests.\n",
    "\n",
    ":::tip\n",
    "By inherting from `BaseRetriever`, your retriever automatically becomes a LangChain [Runnable](/docs/concepts#interface) and will gain the standard `Runnable` functionality out of the box!\n",
    ":::\n",
    "\n",
    "\n",
    ":::info\n",
    "You can use a `RunnableLambda` or `RunnableGenerator` to implement a retriever.\n",
    "\n",
    "The main benefit of implementing a retriever as a `BaseRetriever` vs. a `RunnableLambda` (a custom [runnable function](/docs/how_to/functions)) is that a `BaseRetriever` is a well\n",
    "known LangChain entity so some tooling for monitoring may implement specialized behavior for retrievers. Another difference\n",
    "is that a `BaseRetriever` will behave slightly differently from `RunnableLambda` in some APIs; e.g., the `start` event\n",
    "in `astream_events` API will be `on_retriever_start` instead of `on_chain_start`.\n",
    ":::\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2be9fe82-0757-41d1-a647-15bed11fd3bf",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "Let's implement a toy retriever that returns all documents whose text contains the text in the user query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bdf61902-2984-493b-a002-d4fced6df590",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from langchain_core.callbacks import CallbackManagerForRetrieverRun\n",
    "from langchain_core.documents import Document\n",
    "from langchain_core.retrievers import BaseRetriever\n",
    "\n",
    "\n",
    "class ToyRetriever(BaseRetriever):\n",
    "    \"\"\"A toy retriever that contains the top k documents that contain the user query.\n",
    "\n",
    "    This retriever only implements the sync method _get_relevant_documents.\n",
    "\n",
    "    If the retriever were to involve file access or network access, it could benefit\n",
    "    from a native async implementation of `_aget_relevant_documents`.\n",
    "\n",
    "    As usual, with Runnables, there's a default async implementation that's provided\n",
    "    that delegates to the sync implementation running on another thread.\n",
    "    \"\"\"\n",
    "\n",
    "    documents: List[Document]\n",
    "    \"\"\"List of documents to retrieve from.\"\"\"\n",
    "    k: int\n",
    "    \"\"\"Number of top results to return\"\"\"\n",
    "\n",
    "    def _get_relevant_documents(\n",
    "        self, query: str, *, run_manager: CallbackManagerForRetrieverRun\n",
    "    ) -> List[Document]:\n",
    "        \"\"\"Sync implementations for retriever.\"\"\"\n",
    "        matching_documents = []\n",
    "        for document in documents:\n",
    "            if len(matching_documents) > self.k:\n",
    "                return matching_documents\n",
    "\n",
    "            if query.lower() in document.page_content.lower():\n",
    "                matching_documents.append(document)\n",
    "        return matching_documents\n",
    "\n",
    "    # Optional: Provide a more efficient native implementation by overriding\n",
    "    # _aget_relevant_documents\n",
    "    # async def _aget_relevant_documents(\n",
    "    #     self, query: str, *, run_manager: AsyncCallbackManagerForRetrieverRun\n",
    "    # ) -> List[Document]:\n",
    "    #     \"\"\"Asynchronously get documents relevant to a query.\n",
    "\n",
    "    #     Args:\n",
    "    #         query: String to find relevant documents for\n",
    "    #         run_manager: The callbacks handler to use\n",
    "\n",
    "    #     Returns:\n",
    "    #         List of relevant documents\n",
    "    #     \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eac1f28-29c1-4888-b3aa-b4fa70c73b4c",
   "metadata": {},
   "source": [
    "## Test it 🧪"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ea868db5-48cc-4ec2-9b0a-1ab94c32b302",
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = [\n",
    "    Document(\n",
    "        page_content=\"Dogs are great companions, known for their loyalty and friendliness.\",\n",
    "        metadata={\"type\": \"dog\", \"trait\": \"loyalty\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Cats are independent pets that often enjoy their own space.\",\n",
    "        metadata={\"type\": \"cat\", \"trait\": \"independence\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Goldfish are popular pets for beginners, requiring relatively simple care.\",\n",
    "        metadata={\"type\": \"fish\", \"trait\": \"low maintenance\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Parrots are intelligent birds capable of mimicking human speech.\",\n",
    "        metadata={\"type\": \"bird\", \"trait\": \"intelligence\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Rabbits are social animals that need plenty of space to hop around.\",\n",
    "        metadata={\"type\": \"rabbit\", \"trait\": \"social\"},\n",
    "    ),\n",
    "]\n",
    "retriever = ToyRetriever(documents=documents, k=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "18be85e9-6ef0-4ee0-ae5d-a0810c38b254",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Cats are independent pets that often enjoy their own space.', metadata={'type': 'cat', 'trait': 'independence'}),\n",
       " Document(page_content='Rabbits are social animals that need plenty of space to hop around.', metadata={'type': 'rabbit', 'trait': 'social'})]"
      ]
     },
     "execution_count": 22,
     "metadata": {},


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e5715368",
   "metadata": {},
   "source": [
    "# How to track token usage in ChatModels\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "- [Chat models](/docs/concepts/#chat-models)\n",
    "\n",
    ":::\n",
    "\n",
    "Tracking token usage to calculate cost is an important part of putting your app in production. This guide goes over how to obtain this information from your LangChain model calls.\n",
    "\n",
    "This guide requires `langchain-openai >= 0.1.9`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c7d1338-dd1b-4d06-b33d-d5cffc49fd6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet langchain langchain-openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "598ae1e2-a52d-4459-81fd-cdc68b06742a",
   "metadata": {},
   "source": [
    "## Using LangSmith\n",
    "\n",
    "You can use [LangSmith](https://www.langchain.com/langsmith) to help track token usage in your LLM application. See the [LangSmith quick start guide](https://docs.smith.langchain.com/).\n",
    "\n",
    "## Using AIMessage.usage_metadata\n",
    "\n",
    "A number of model providers return token usage information as part of the chat generation response. When available, this information will be included on the `AIMessage` objects produced by the corresponding model.\n",
    "\n",
    "LangChain `AIMessage` objects include a [usage_metadata](https://python.langchain.com/api_reference/core/messages/langchain_core.messages.ai.AIMessage.html#langchain_core.messages.ai.AIMessage.usage_metadata) attribute. When populated, this attribute will be a [UsageMetadata](https://python.langchain.com/api_reference/core/messages/langchain_core.messages.ai.UsageMetadata.html) dictionary with standard keys (e.g., `\"input_tokens\"` and `\"output_tokens\"`).\n",
    "\n",
    "Examples:\n",
    "\n",
    "**OpenAI**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b39bf807-4125-4db4-bbf7-28a46afff6b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_tokens': 8, 'output_tokens': 9, 'total_tokens': 17}"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# # !pip install -qU langchain-openai\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
    "openai_response = llm.invoke(\"hello\")\n",
    "openai_response.usage_metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2299c44a-2fe6-4d52-a6a2-99ff6d231c73",
   "metadata": {},
   "source": [
    "**Anthropic**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9c82ff80-ec4e-4049-b019-5f0bbd7df82a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_tokens': 8, 'output_tokens': 12, 'total_tokens': 20}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# !pip install -qU langchain-anthropic\n",
    "\n",
    "from langchain_anthropic import ChatAnthropic\n",
    "\n",
    "llm = ChatAnthropic(model=\"claude-3-haiku-20240307\")\n",
    "anthropic_response = llm.invoke(\"hello\")\n",
    "anthropic_response.usage_metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d4efc15-ba9f-4b3d-9278-8e01f99f263f",
   "metadata": {},
   "source": [
    "### Using AIMessage.response_metadata\n",
    "\n",
    "Metadata from the model response is also included in the AIMessage [response_metadata](https://python.langchain.com/api_reference/core/messages/langchain_core.messages.ai.AIMessage.html#langchain_core.messages.ai.AIMessage.response_metadata) attribute. These data are typically not standardized. Note that different providers adopt different conventions for representing token counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f156f9da-21f2-4c81-a714-54cbf9ad393e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OpenAI: {'completion_tokens': 9, 'prompt_tokens': 8, 'total_tokens': 17}\n",
      "\n",
      "Anthropic: {'input_tokens': 8, 'output_tokens': 12}\n"
     ]
    }
   ],
   "source": [
    "print(f'OpenAI: {openai_response.response_metadata[\"token_usage\"]}\\n')\n",
    "print(f'Anthropic: {anthropic_response.response_metadata[\"usage\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ef2c43-0ff6-49eb-9782-e4070c9da8d7",
   "metadata": {},
   "source": [
    "### Streaming\n",
    "\n",
    "Some providers support token count metadata in a streaming context.\n",
    "\n",
    "#### OpenAI\n",
    "\n",
    "For example, OpenAI will return a message [chunk](https://python.langchain.com/api_reference/core/messages/langchain_core.messages.ai.AIMessageChunk.html) at the end of a stream with token usage information. This behavior is supported by `langchain-openai >= 0.1.9` and can be enabled by setting `stream_usage=True`. This attribute can also be set when `ChatOpenAI` is instantiated.\n",
    "\n",
    ":::note\n",
    "By default, the last message chunk in a stream will include a `\"finish_reason\"` in the message's `response_metadata` attribute. If we include token usage in streaming mode, an additional chunk containing usage metadata will be added to the end of the stream, such that `\"finish_reason\"` appears on the second to last message chunk.\n",
    ":::\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "07f0c872-6b6c-4fed-a129-9b5a858505be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content='' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content='Hello' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content='!' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content=' How' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content=' can' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content=' I' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content=' assist' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",
      "content=' you' id='run-adb20c31-60c7-43a2-99b2-d4a53ca5f623'\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b8982428",
   "metadata": {},
   "source": [
    "# Run models locally\n",
    "\n",
    "## Use case\n",
    "\n",
    "The popularity of projects like [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://github.com/ollama/ollama), [GPT4All](https://github.com/nomic-ai/gpt4all), [llamafile](https://github.com/Mozilla-Ocho/llamafile), and others underscore the demand to run LLMs locally (on your own device).\n",
    "\n",
    "This has at least two important benefits:\n",
    "\n",
    "1. `Privacy`: Your data is not sent to a third party, and it is not subject to the terms of service of a commercial service\n",
    "2. `Cost`: There is no inference fee, which is important for token-intensive applications (e.g., [long-running simulations](https://twitter.com/RLanceMartin/status/1691097659262820352?s=20), summarization)\n",
    "\n",
    "## Overview\n",
    "\n",
    "Running an LLM locally requires a few things:\n",
    "\n",
    "1. `Open-source LLM`: An open-source LLM that can be freely modified and shared \n",
    "2. `Inference`: Ability to run this LLM on your device w/ acceptable latency\n",
    "\n",
    "### Open-source LLMs\n",
    "\n",
    "Users can now gain access to a rapidly growing set of [open-source LLMs](https://cameronrwolfe.substack.com/p/the-history-of-open-source-llms-better). \n",
    "\n",
    "These LLMs can be assessed across at least two dimensions (see figure):\n",
    " \n",
    "1. `Base model`: What is the base-model and how was it trained?\n",
    "2. `Fine-tuning approach`: Was the base-model fine-tuned and, if so, what [set of instructions](https://cameronrwolfe.substack.com/p/beyond-llama-the-power-of-open-llms#%C2%A7alpaca-an-instruction-following-llama-model) was used?\n",
    "\n",
    "![Image description](../../static/img/OSS_LLM_overview.png)\n",
    "\n",
    "The relative performance of these models can be assessed using several leaderboards, including:\n",
    "\n",
    "1. [LmSys](https://chat.lmsys.org/?arena)\n",
    "2. [GPT4All](https://gpt4all.io/index.html)\n",
    "3. [HuggingFace](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard)\n",
    "\n",
    "### Inference\n",
    "\n",
    "A few frameworks for this have emerged to support inference of open-source LLMs on various devices:\n",
    "\n",
    "1. [`llama.cpp`](https://github.com/ggerganov/llama.cpp): C++ implementation of llama inference code with [weight optimization / quantization](https://finbarr.ca/how-is-llama-cpp-possible/)\n",
    "2. [`gpt4all`](https://docs.gpt4all.io/index.html): Optimized C backend for inference\n",
    "3. [`Ollama`](https://ollama.ai/): Bundles model weights and environment into an app that runs on device and serves the LLM\n",
    "4. [`llamafile`](https://github.com/Mozilla-Ocho/llamafile): Bundles model weights and everything needed to run the model in a single file, allowing you to run the LLM locally from this file without any additional installation steps\n",
    "\n",
    "In general, these frameworks will do a few things:\n",
    "\n",
    "1. `Quantization`: Reduce the memory footprint of the raw model weights\n",
    "2. `Efficient implementation for inference`: Support inference on consumer hardware (e.g., CPU or laptop GPU)\n",
    "\n",
    "In particular, see [this excellent post](https://finbarr.ca/how-is-llama-cpp-possible/) on the importance of quantization.\n",
    "\n",
    "![Image description](../../static/img/llama-memory-weights.png)\n",
    "\n",
    "With less precision, we radically decrease the memory needed to store the LLM in memory.\n",
    "\n",
    "In addition, we can see the importance of GPU memory bandwidth [sheet](https://docs.google.com/spreadsheets/d/1OehfHHNSn66BP2h3Bxp2NJTVX97icU0GmCXF6pK23H8/edit#gid=0)!\n",
    "\n",
    "A Mac M2 Max is 5-6x faster than a M1 for inference due to the larger GPU memory bandwidth.\n",
    "\n",
    "![Image description](../../static/img/llama_t_put.png)\n",
    "\n",
    "### Formatting prompts\n",
    "\n",
    "Some providers have [chat model](/docs/concepts/#chat-models) wrappers that takes care of formatting your input prompt for the specific local model you're using. However, if you are prompting local models with a [text-in/text-out LLM](/docs/concepts/#llms) wrapper, you may need to use a prompt tailed for your specific model.\n",
    "\n",
    "This can [require the inclusion of special tokens](https://huggingface.co/blog/llama2#how-to-prompt-llama-2). [Here's an example for LLaMA 2](https://smith.langchain.com/hub/rlm/rag-prompt-llama).\n",
    "\n",
    "## Quickstart\n",
    "\n",
    "[`Ollama`](https://ollama.ai/) is one way to easily run inference on macOS.\n",
    " \n",
    "The instructions [here](https://github.com/jmorganca/ollama?tab=readme-ov-file#ollama) provide details, which we summarize:\n",
    " \n",
    "* [Download and run](https://ollama.ai/download) the app\n",
    "* From command line, fetch a model from this [list of options](https://github.com/jmorganca/ollama): e.g., `ollama pull llama3.1:8b`\n",
    "* When the app is running, all models are automatically served on `localhost:11434`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29450fc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain_ollama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "86178adb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'...Neil Armstrong!\\n\\nOn July 20, 1969, Neil Armstrong became the first person to set foot on the lunar surface, famously declaring \"That\\'s one small step for man, one giant leap for mankind\" as he stepped off the lunar module Eagle onto the Moon\\'s surface.\\n\\nWould you like to know more about the Apollo 11 mission or Neil Armstrong\\'s achievements?'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_ollama import OllamaLLM\n",
    "\n",
    "llm = OllamaLLM(model=\"llama3.1:8b\")\n",
    "\n",
    "llm.invoke(\"The first man on the moon was ...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "674cc672",
   "metadata": {},
   "source": [
    "Stream tokens as they are being generated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1386a852",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...|"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [


"Neil| Armstrong|,| an| American| astronaut|.| He| stepped| out| of| the| lunar| module| Eagle| and| onto| the| surface| of| the| Moon| on| July| |20|,| |196|9|,| famously| declaring|:| \"|That|'s| one| small| step| for| man|,| one| giant| leap| for| mankind|.\"||"
     ]
    }
   ],
   "source": [
    "for chunk in llm.stream(\"The first man on the moon was ...\"):\n",
    "    print(chunk, end=\"|\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5731060",
   "metadata": {},
   "source": [
    "Ollama also includes a chat model wrapper that handles formatting conversation turns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f14a778a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='The answer is a historic one!\\n\\nThe first man to walk on the Moon was Neil Armstrong, an American astronaut and commander of the Apollo 11 mission. On July 20, 1969, Armstrong stepped out of the lunar module Eagle onto the surface of the Moon, famously declaring:\\n\\n\"That\\'s one small step for man, one giant leap for mankind.\"\\n\\nArmstrong was followed by fellow astronaut Edwin \"Buzz\" Aldrin, who also walked on the Moon during the mission. Michael Collins remained in orbit around the Moon in the command module Columbia.\\n\\nNeil Armstrong passed away on August 25, 2012, but his legacy as a pioneering astronaut and engineer continues to inspire people around the world!', response_metadata={'model': 'llama3.1:8b', 'created_at': '2024-08-01T00:38:29.176717Z', 'message': {'role': 'assistant', 'content': ''}, 'done_reason': 'stop', 'done': True, 'total_duration': 10681861417, 'load_duration': 34270292, 'prompt_eval_count': 19, 'prompt_eval_duration': 6209448000, 'eval_count': 141, 'eval_duration': 4432022000}, id='run-7bed57c5-7f54-4092-912c-ae49073dcd48-0', usage_metadata={'input_tokens': 19, 'output_tokens': 141, 'total_tokens': 160})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_ollama import ChatOllama\n",
    "\n",
    "chat_model = ChatOllama(model=\"llama3.1:8b\")\n",
    "\n",
    "chat_model.invoke(\"Who was the first man on the moon?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb27414",
   "metadata": {},
   "source": [
    "## Environment\n",
    "\n",
    "Inference speed is a challenge when running models locally (see above).\n",
    "\n",
    "To minimize latency, it is desirable to run models locally on GPU, which ships with many consumer laptops [e.g., Apple devices](https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/).\n",
    "\n",
    "And even with GPU, the available GPU memory bandwidth (as noted above) is important.\n",
    "\n",
    "### Running Apple silicon GPU\n",
    "\n",
    "`Ollama` and [`llamafile`](https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file#gpu-support) will automatically utilize the GPU on Apple devices.\n",
    " \n",
    "Other frameworks require the user to set up the environment to utilize the Apple GPU.\n",
    "\n",
    "For example, `llama.cpp` python bindings can be configured to use the GPU via [Metal](https://developer.apple.com/metal/).\n",
    "\n",
    "Metal is a graphics and compute API created by Apple providing near-direct access to the GPU. \n",
    "\n",
    "See the [`llama.cpp`](/docs/integrations/llms/llamacpp) setup [here](https://github.com/abetlen/llama-cpp-python/blob/main/docs/install/macos.md) to enable this.\n",
    "\n",
    "In particular, ensure that conda is using the correct virtual environment that you created (`miniforge3`).\n",
    "\n",
    "E.g., for me:\n",
    "\n",
    "```\n",
    "conda activate /Users/rlm/miniforge3/envs/llama\n",
    "```\n",
    "\n",
    "With the above confirmed, then:\n",
    "\n",
    "```\n",
    "CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c382e79a",
   "metadata": {},
   "source": [
    "## LLMs\n",
    "\n",
    "There are various ways to gain access to quantized model weights.\n",
    "\n",
    "1. [`HuggingFace`](https://huggingface.co/TheBloke) - Many quantized model are available for download and can be run with framework such as [`llama.cpp`](https://github.com/ggerganov/llama.cpp). You can also download models in [`llamafile` format](https://huggingface.co/models?other=llamafile) from HuggingFace.\n",
    "2. [`gpt4all`](https://gpt4all.io/index.html) - The model explorer offers a leaderboard of metrics and associated quantized models available for download \n",
    "3. [`Ollama`](https://github.com/jmorganca/ollama) - Several models can be accessed directly via `pull`\n",
    "\n",
    "### Ollama\n",
    "\n",
    "With [Ollama](https://github.com/jmorganca/ollama), fetch a model via `ollama pull <model family>:<tag>`:\n",
    "\n",
    "* E.g., for Llama 2 7b: `ollama pull llama2` will download the most basic version of the model (e.g., smallest # parameters and 4 bit quantization)\n",
    "* We can also specify a particular version from the [model list](https://github.com/jmorganca/ollama?tab=readme-ov-file#model-library), e.g., `ollama pull llama2:13b`\n",
    "* See the full set of parameters on the [API reference page](https://python.langchain.com/api_reference/community/llms/langchain_community.llms.ollama.Ollama.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "8ecd2f78",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' Sure! Here\\'s the answer, broken down step by step:\\n\\nThe first man on the moon was... Neil Armstrong.\\n\\nHere\\'s how I arrived at that answer:\\n\\n1. The first manned mission to land on the moon was Apollo 11.\\n2. The mission included three astronauts: Neil Armstrong, Edwin \"Buzz\" Aldrin, and Michael Collins.\\n3. Neil Armstrong was the mission commander and the first person to set foot on the moon.\\n4. On July 20, 1969, Armstrong stepped out of the lunar module Eagle and onto the moon\\'s surface, famously declaring \"That\\'s one small step for man, one giant leap for mankind.\"\\n\\nSo, the first man on the moon was Neil Armstrong!'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm = OllamaLLM(model=\"llama2:13b\")\n",
    "llm.invoke(\"The first man on the moon was ... think step by step\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07c8c0d1",
   "metadata": {},
   "source": [
    "### Llama.cpp\n",
    "\n",
    "Llama.cpp is compatible with a [broad set of models](https://github.com/ggerganov/llama.cpp).\n",
    "\n",