{
 "cells": [
  {
   "cell_type": "raw",
   "id": "04171ad7",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "keywords: [custom tool, custom tools]\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5436020b",
   "metadata": {},
   "source": [
    "# How to create Tools\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [LangChain tools](/docs/concepts#tools)\n",
    "- [Agents](/docs/concepts/#agents)\n",
    "\n",
    ":::\n",
    "\n",
    "When constructing your own agent, you will need to provide it with a list of Tools that it can use. While LangChain includes some prebuilt tools, it can often be more useful to use tools that use custom logic. This guide will walk you through some ways you can create custom tools.\n",
    "\n",
    "The biggest difference here is that the first function requires an object with multiple input fields, while the second one only accepts an object with a single field. Some older agents only work with functions that require single inputs, so it's important to understand the distinction.\n",
    "\n",
    "LangChain has a handful of ways to construct tools for different applications. Below I'll show the two most common ways to create tools, and where you might use each."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82bb159d",
   "metadata": {},
   "source": [
    "## Tool schema\n",
    "\n",
    "```{=mdx}\n",
    ":::caution Compatibility\n",
    "Only available in `@langchain/core` version 0.2.19 and above.\n",
    ":::\n",
    "```\n",
    "\n",
    "The simplest way to create a tool is through the [`StructuredToolParams`](https://api.js.langchain.com/interfaces/_langchain_core.tools.StructuredToolParams.html) schema. Every chat model which supports tool calling in LangChain accepts binding tools to the model through this schema. This schema has only three fields\n",
    "\n",
    "- `name` - The name of the tool.\n",
    "- `schema` - The schema of the tool, defined with a Zod object.\n",
    "- `description` (optional) - A description of the tool.\n",
    "\n",
    "This schema does not include a function to pair with the tool, and for this reason it should only be used in situations where the generated output does not need to be passed as the input argument to a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d129789",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { z } from \"zod\";\n",
    "import { StructuredToolParams } from \"@langchain/core/tools\";\n",
    "\n",
    "const simpleToolSchema: StructuredToolParams = {\n",
    "  name: \"get_current_weather\",\n",
    "  description: \"Get the current weather for a location\",\n",
    "  schema: z.object({\n",
    "    city: z.string().describe(\"The city to get the weather for\"),\n",
    "    state: z.string().optional().describe(\"The state to get the weather for\"),\n",
    "  })\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6ec6ee8",
   "metadata": {},
   "source": [
    "## `tool` function\n",
    "\n",
    "```{=mdx}\n",
    ":::caution Compatibility\n",
    "Only available in `@langchain/core` version 0.2.7 and above.\n",
    ":::\n",
    "```\n",
    "\n",
    "The [`tool`](https://api.js.langchain.com/classes/langchain_core.tools.Tool.html) wrapper function is a convenience method for turning a JavaScript function into a tool. It requires the function itself along with some additional arguments that define your tool. You should use this over `StructuredToolParams` tools when the resulting tool call executes a function. The most important are:\n",
    "\n",
    "- The tool's `name`, which the LLM will use as context as well as to reference the tool\n",
    "- An optional, but recommended `description`, which the LLM will use as context to know when to use the tool\n",
    "- A `schema`, which defines the shape of the tool's input\n",
    "\n",
    "The `tool` function will return an instance of the [`StructuredTool`](https://api.js.langchain.com/classes/langchain_core.tools.StructuredTool.html) class, so it is compatible with all the existing tool calling infrastructure in the LangChain library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ecc1ce9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32m\"The sum of 1 and 2 is 3\"\u001b[39m"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import { z } from \"zod\";\n",
    "import { tool } from \"@langchain/core/tools\";\n",
    "\n",
    "const adderSchema = z.object({\n",
    "  a: z.number(),\n",
    "  b: z.number(),\n",
    "});\n",
    "const adderTool = tool(async (input): Promise<string> => {\n",
    "  const sum = input.a + input.b;\n",
    "  return `The sum of ${input.a} and ${input.b} is ${sum}`;\n",
    "}, {\n",
    "  name: \"adder\",\n",
    "  description: \"Adds two numbers together\",\n",
    "  schema: adderSchema,\n",
    "});\n",
    "\n",
    "await adderTool.invoke({ a: 1, b: 2 });"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "213ee344",
   "metadata": {},
   "source": [
    "## `DynamicStructuredTool`\n",
    "\n",
    "You can also use the [`DynamicStructuredTool`](https://api.js.langchain.com/classes/langchain_core.tools.DynamicStructuredTool.html) class to declare tools. Here's an example - note that tools must always return strings!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "833dda4a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32m\"72\"\u001b[39m"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import { DynamicStructuredTool } from \"@langchain/core/tools\";\n",
    "import { z } from \"zod\";\n",
    "\n",
    "const multiplyTool = new DynamicStructuredTool({\n",
    "  name: \"multiply\",\n",
    "  description: \"multiply two numbers together\",\n",
    "  schema: z.object({\n",
    "    a: z.number().describe(\"the first number to multiply\"),\n",
    "    b: z.number().describe(\"the second number to multiply\"),\n",
    "  }),\n",
    "  func: async ({ a, b }: { a: number; b: number; }) => {\n",
    "    return (a * b).toString();\n",
    "  },\n",
    "});\n",
    "\n",
    "await multiplyTool.invoke({ a: 8, b: 9, });"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7326b23",
   "metadata": {},
   "source": [
    "## `DynamicTool`\n",
    "\n",
    "For older agents that require tools which accept only a single input, you can pass the relevant parameters to the [`DynamicTool`](https://api.js.langchain.com/classes/langchain_core.tools.DynamicTool.html) class. This is useful when working with older agents that only support tools that accept a single input. In this case, no schema is required:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b0ce7de8",
   "metadata": {},
   "outputs": [
    {


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "beba2e0e",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 2\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb0735c0",
   "metadata": {},
   "source": [
    "# How to use few shot examples in chat models\n",
    "\n",
    "This guide covers how to prompt a chat model with example inputs and outputs. Providing the model with a few such examples is called few-shotting, and is a simple yet powerful way to guide generation and in some cases drastically improve model performance.\n",
    "\n",
    "There does not appear to be solid consensus on how best to do few-shot prompting, and the optimal prompt compilation will likely vary by model. Because of this, we provide few-shot prompt templates like the [FewShotChatMessagePromptTemplate](https://api.js.langchain.com/classes/langchain_core.prompts.FewShotChatMessagePromptTemplate.html) as a flexible starting point, and you can modify or replace them as you see fit.\n",
    "\n",
    "The goal of few-shot prompt templates are to dynamically select examples based on an input, and then format the examples in a final prompt to provide for the model.\n",
    "\n",
    "**Note:** The following code examples are for chat models only, since `FewShotChatMessagePromptTemplates` are designed to output formatted [chat messages](/docs/concepts/#message-types) rather than pure strings. For similar few-shot prompt examples for pure string templates compatible with completion models (LLMs), see the [few-shot prompt templates](/docs/how_to/few_shot_examples/) guide.\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [Prompt templates](/docs/concepts/#prompt-templates)\n",
    "- [Example selectors](/docs/concepts/#example-selectors)\n",
    "- [Chat models](/docs/concepts/#chat-model)\n",
    "- [Vectorstores](/docs/concepts/#vectorstores)\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d716f2de-cc29-4823-9360-a808c7bfdb86",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Fixed Examples\n",
    "\n",
    "The most basic (and common) few-shot prompting technique is to use fixed prompt examples. This way you can select a chain, evaluate it, and avoid worrying about additional moving parts in production.\n",
    "\n",
    "The basic components of the template are:\n",
    "- `examples`: An array of object examples to include in the final prompt.\n",
    "- `examplePrompt`: converts each example into 1 or more messages through its [`formatMessages`](https://api.js.langchain.com/classes/langchain_core.prompts.FewShotChatMessagePromptTemplate.html#formatMessages) method. A common example would be to convert each example into one human message and one AI message response, or a human message followed by a function call message.\n",
    "\n",
    "Below is a simple demonstration. First, define the examples you'd like to include:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0fc5a02a-6249-4e92-95c3-30fff9671e8b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import {\n",
    "    ChatPromptTemplate,\n",
    "    FewShotChatMessagePromptTemplate,\n",
    "} from \"@langchain/core/prompts\"\n",
    "\n",
    "const examples = [\n",
    "    { input: \"2+2\", output: \"4\" },\n",
    "    { input: \"2+3\", output: \"5\" },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8710ecc-2aa0-4172-a74c-250f6bc3d9e2",
   "metadata": {},
   "source": [
    "Next, assemble them into the few-shot prompt template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "65e72ad1-9060-47d0-91a1-bc130c8b98ac",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "  HumanMessage {\n",
      "    lc_serializable: true,\n",
      "    lc_kwargs: { content: \"2+2\", additional_kwargs: {}, response_metadata: {} },\n",
      "    lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "    content: \"2+2\",\n",
      "    name: undefined,\n",
      "    additional_kwargs: {},\n",
      "    response_metadata: {}\n",
      "  },\n",
      "  AIMessage {\n",
      "    lc_serializable: true,\n",
      "    lc_kwargs: {\n",
      "      content: \"4\",\n",
      "      tool_calls: [],\n",
      "      invalid_tool_calls: [],\n",
      "      additional_kwargs: {},\n",
      "      response_metadata: {}\n",
      "    },\n",
      "    lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "    content: \"4\",\n",
      "    name: undefined,\n",
      "    additional_kwargs: {},\n",
      "    response_metadata: {},\n",
      "    tool_calls: [],\n",
      "    invalid_tool_calls: []\n",
      "  },\n",
      "  HumanMessage {\n",
      "    lc_serializable: true,\n",
      "    lc_kwargs: { content: \"2+3\", additional_kwargs: {}, response_metadata: {} },\n",
      "    lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "    content: \"2+3\",\n",
      "    name: undefined,\n",
      "    additional_kwargs: {},\n",
      "    response_metadata: {}\n",
      "  },\n",
      "  AIMessage {\n",
      "    lc_serializable: true,\n",
      "    lc_kwargs: {\n",
      "      content: \"5\",\n",
      "      tool_calls: [],\n",
      "      invalid_tool_calls: [],\n",
      "      additional_kwargs: {},\n",
      "      response_metadata: {}\n",
      "    },\n",
      "    lc_namespace: [ \"langchain_core\", \"messages\" ],\n",
      "    content: \"5\",\n",
      "    name: undefined,\n",
      "    additional_kwargs: {},\n",
      "    response_metadata: {},\n",
      "    tool_calls: [],\n",
      "    invalid_tool_calls: []\n",
      "  }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "// This is a prompt template used to format each individual example.\n",
    "const examplePrompt = ChatPromptTemplate.fromMessages(\n",
    "    [\n",
    "        [\"human\", \"{input}\"],\n",
    "        [\"ai\", \"{output}\"],\n",
    "    ]\n",
    ")\n",
    "const fewShotPrompt = new FewShotChatMessagePromptTemplate({\n",
    "    examplePrompt,\n",
    "    examples,\n",
    "    inputVariables: [], // no input variables\n",
    "})\n",
    "\n",
    "const result = await fewShotPrompt.invoke({});\n",
    "console.log(result.toChatMessages())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5490bd59-b28f-46a4-bbdf-0191802dd3c5",
   "metadata": {},
   "source": [
    "Finally, we assemble the final prompt as shown below, passing `fewShotPrompt` directly into the `fromMessages` factory method, and use it with a model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9f86d6d9-50de-41b6-b6c7-0f9980cc0187",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "const finalPrompt = ChatPromptTemplate.fromMessages(\n",
    "    [\n",
    "        [\"system\", \"You are a wondrous wizard of math.\"],\n",
    "        fewShotPrompt,\n",


# How to invoke runnables in parallel

:::info Prerequisites

This guide assumes familiarity with the following concepts:

- [LangChain Expression Language (LCEL)](/docs/concepts/#langchain-expression-language)
- [Chaining runnables](/docs/how_to/sequence/)

:::

The [`RunnableParallel`](https://api.js.langchain.com/classes/langchain_core.runnables.RunnableParallel.html) (also known as a `RunnableMap`) primitive is an object whose values are runnables (or things that can be coerced to runnables, like functions).
It runs all of its values in parallel, and each value is called with the initial input to the `RunnableParallel`. The final return value is an object with the results of each value under its appropriate key.

## Formatting with `RunnableParallels`

`RunnableParallels` are useful for parallelizing operations, but can also be useful for manipulating the output of one Runnable to match the input format of the next Runnable in a sequence. You can use them to split or fork the chain so that multiple components can process the input in parallel. Later, other components can join or merge the results to synthesize a final response. This type of chain creates a computation graph that looks like the following:

```text
     Input
      / \
     /   \
 Branch1 Branch2
     \   /
      \ /
      Combine
```

Below, the input to each chain in the `RunnableParallel` is expected to be an object with a key for `"topic"`.
We can satisfy that requirement by invoking our chain with an object matching that structure.

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm install @langchain/anthropic @langchain/cohere @langchain/core
```

import CodeBlock from "@theme/CodeBlock";
import BasicExample from "@examples/guides/expression_language/runnable_maps_basic.ts";

<CodeBlock language="typescript">{BasicExample}</CodeBlock>

## Manipulating outputs/inputs

Maps can be useful for manipulating the output of one Runnable to match the input format of the next Runnable in a sequence.

Note below that the object within the `RunnableSequence.from()` call is automatically coerced into a runnable map. All keys of the object must
have values that are runnables or can be themselves coerced to runnables (functions to `RunnableLambda`s or objects to `RunnableMap`s).
This coercion will also occur when composing chains via the `.pipe()` method.

import SequenceExample from "@examples/guides/expression_language/runnable_maps_sequence.ts";

<CodeBlock language="typescript">{SequenceExample}</CodeBlock>

Here the input to prompt is expected to be a map with keys "context" and "question". The user input is just the question. So we need to get the context using our retriever and passthrough the user input under the "question" key.

## Next steps

You now know some ways to format and parallelize chain steps with `RunnableParallel`.

Next, you might be interested in [using custom logic](/docs/how_to/functions/) in your chains.


While some model providers support [built-in ways to return structured output](/docs/how_to/structured_output), not all do. We can use an output parser to help users to specify an arbitrary JSON schema via the prompt, query a model for outputs that conform to that schema, and finally parse that schema as JSON.

Keep in mind that large language models are leaky abstractions! You'll have to use an LLM with sufficient capacity to generate well-formed JSON.

This guide assumes familiarity with the following concepts:

- [Chaining runnables together](/docs/how_to/sequence/)

The [`JsonOutputParser`](https://api.js.langchain.com/classes/langchain_core.output_parsers.JsonOutputParser.html) is one built-in option for prompting for and then parsing JSON output.

import ChatModelTabs from "@theme/ChatModelTabs";

setup: "Why don't scientists trust atoms?",

punchline: "Because they make up everything!"

import { ChatOpenAI } from "@langchain/openai";

import { JsonOutputParser } from "@langchain/core/output_parsers"

import { ChatPromptTemplate } from "@langchain/core/prompts"

// Define your desired data structure. Only used for typing the parser output.

// A query and format instructions used to prompt a language model.

const formatInstructions = "Respond with a valid JSON object, containing two fields: 'setup' and 'punchline'."

// Set up a parser + inject instructions into the prompt template.

const parser = new JsonOutputParser<Joke>()

const prompt = ChatPromptTemplate.fromTemplate(

"Answer the user query.\n{format_instructions}\n{query}\n"

const partialedPrompt = await prompt.partial({

const chain = partialedPrompt.pipe(model).pipe(parser);

await chain.invoke({ query: jokeQuery });

The `JsonOutputParser` also supports streaming partial chunks. This is useful when the model returns partial JSON output in multiple chunks. The parser will keep track of the partial chunks and return the final JSON output when the model finishes generating the output.

stdout

{ setup: "" }

{ setup: "Why" }

{ setup: "Why don't" }

{ setup: "Why don't scientists" }

{ setup: "Why don't scientists trust" }

{ setup: "Why don't scientists trust atoms" }

{ setup: "Why don't scientists trust atoms?", punchline: "" }

{ setup: "Why don't scientists trust atoms?", punchline: "Because" }

setup: "Why don't scientists trust atoms?",

punchline: "Because they"

setup: "Why don't scientists trust atoms?",

punchline: "Because they make"

setup: "Why don't scientists trust atoms?",

punchline: "Because they make up"

setup: "Why don't scientists trust atoms?",

punchline: "Because they make up everything"

setup: "Why don't scientists trust atoms?",

punchline: "Because they make up everything!"

for await (const s of await chain.stream({ query: jokeQuery })) {

You've now learned one way to prompt a model to return structured JSON. Next, check out the [broader guide on obtaining structured output](/docs/how_to/structured_output) for other techniques.


---
keywords: [similaritySearchWithScore]
---

# How to create and query vector stores

:::info
Head to [Integrations](/docs/integrations/vectorstores) for documentation on built-in integrations with vectorstore providers.
:::

:::info Prerequisites

This guide assumes familiarity with the following concepts:

- [Vector stores](/docs/concepts/#vectorstores)
- [Embeddings](/docs/concepts/#embedding-models)
- [Document loaders](/docs/concepts#document-loaders)

:::

One of the most common ways to store and search over unstructured data is to embed it and store the resulting embedding
vectors, and then at query time to embed the unstructured query and retrieve the embedding vectors that are
'most similar' to the embedded query. A vector store takes care of storing embedded data and performing vector search
for you.

This walkthrough uses a basic, unoptimized implementation called [`MemoryVectorStore`](https://api.js.langchain.com/classes/langchain.vectorstores_memory.MemoryVectorStore.html) that stores embeddings in-memory and does an exact, linear search for the most similar embeddings.
LangChain contains many built-in integrations - see [this section](/docs/how_to/vectorstores/#which-one-to-pick) for more, or the [full list of integrations](/docs/integrations/vectorstores/).

## Creating a new index

Most of the time, you'll need to load and prepare the data you want to search over. Here's an example that loads a recent speech from a file:

import ExampleLoader from "@examples/indexes/vector_stores/memory_fromdocs.ts";

<CodeBlock language="typescript">{ExampleLoader}</CodeBlock>

Most of the time, you'll need to split the loaded text as a preparation step. See [this section](/docs/concepts/#text-splitters) to learn more about text splitters.

## Creating a new index from texts

If you have already prepared the data you want to search over, you can initialize a vector store directly from text chunks:

import CodeBlock from "@theme/CodeBlock";
import ExampleTexts from "@examples/indexes/vector_stores/memory.ts";

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm install @langchain/openai @langchain/core
```

<CodeBlock language="typescript">{ExampleTexts}</CodeBlock>

## Which one to pick?

Here's a quick guide to help you pick the right vector store for your use case:

- If you're after something that can just run inside your Node.js application, in-memory, without any other servers to stand up, then go for [HNSWLib](/docs/integrations/vectorstores/hnswlib), [Faiss](/docs/integrations/vectorstores/faiss), [LanceDB](/docs/integrations/vectorstores/lancedb) or [CloseVector](/docs/integrations/vectorstores/closevector)
- If you're looking for something that can run in-memory in browser-like environments, then go for [MemoryVectorStore](/docs/integrations/vectorstores/memory) or [CloseVector](/docs/integrations/vectorstores/closevector)
- If you come from Python and you were looking for something similar to FAISS, try [HNSWLib](/docs/integrations/vectorstores/hnswlib) or [Faiss](/docs/integrations/vectorstores/faiss)
- If you're looking for an open-source full-featured vector database that you can run locally in a docker container, then go for [Chroma](/docs/integrations/vectorstores/chroma)
- If you're looking for an open-source vector database that offers low-latency, local embedding of documents and supports apps on the edge, then go for [Zep](/docs/integrations/vectorstores/zep)
- If you're looking for an open-source production-ready vector database that you can run locally (in a docker container) or hosted in the cloud, then go for [Weaviate](/docs/integrations/vectorstores/weaviate).
- If you're using Supabase already then look at the [Supabase](/docs/integrations/vectorstores/supabase) vector store to use the same Postgres database for your embeddings too
- If you're looking for a production-ready vector store you don't have to worry about hosting yourself, then go for [Pinecone](/docs/integrations/vectorstores/pinecone)
- If you are already utilizing SingleStore, or if you find yourself in need of a distributed, high-performance database, you might want to consider the [SingleStore](/docs/integrations/vectorstores/singlestore) vector store.
- If you are looking for an online MPP (Massively Parallel Processing) data warehousing service, you might want to consider the [AnalyticDB](/docs/integrations/vectorstores/analyticdb) vector store.
- If you're in search of a cost-effective vector database that allows run vector search with SQL, look no further than [MyScale](/docs/integrations/vectorstores/myscale).
- If you're in search of a vector database that you can load from both the browser and server side, check out [CloseVector](/docs/integrations/vectorstores/closevector). It's a vector database that aims to be cross-platform.
- If you're looking for a scalable, open-source columnar database with excellent performance for analytical queries, then consider [ClickHouse](/docs/integrations/vectorstores/clickhouse).

## Next steps

You've now learned how to load data into a vectorstore.

Next, check out the [full tutorial on retrieval-augmented generation](/docs/tutorials/rag).


"data": {
      "text/plain": [
       "{\n",
       "  setup: \u001b[32m\"Why don't cats play poker in the jungle?\"\u001b[39m,\n",
       "  punchline: \u001b[32m\"Too many cheetahs!\"\u001b[39m\n",
       "}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const structuredLlm = model.withStructuredOutput(joke, {\n",
    "  method: \"json_mode\",\n",
    "  name: \"joke\",\n",
    "})\n",
    "\n",
    "await structuredLlm.invoke(\n",
    "  \"Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56278a82",
   "metadata": {},
   "source": [
    "In the above example, we use OpenAI's alternate JSON mode capability along with a more specific prompt.\n",
    "\n",
    "For specifics about the model you choose, peruse its entry in the [API reference pages](https://api.js.langchain.com/).\n",
    "\n",
    "### (Advanced) Raw outputs\n",
    "\n",
    "LLMs aren't perfect at generating structured output, especially as schemas become complex. You can avoid raising exceptions and handle the raw output yourself by passing `includeRaw: true`. This changes the output format to contain the raw message output and the `parsed` value (if successful):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "46b616a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "  raw: AIMessage {\n",
       "    lc_serializable: \u001b[33mtrue\u001b[39m,\n",
       "    lc_kwargs: {\n",
       "      content: \u001b[32m\"\"\u001b[39m,\n",
       "      tool_calls: [\n",
       "        {\n",
       "          name: \u001b[32m\"joke\"\u001b[39m,\n",
       "          args: \u001b[36m[Object]\u001b[39m,\n",
       "          id: \u001b[32m\"call_0pEdltlfSXjq20RaBFKSQOeF\"\u001b[39m\n",
       "        }\n",
       "      ],\n",
       "      invalid_tool_calls: [],\n",
       "      additional_kwargs: { function_call: \u001b[90mundefined\u001b[39m, tool_calls: [ \u001b[36m[Object]\u001b[39m ] },\n",
       "      response_metadata: {}\n",
       "    },\n",
       "    lc_namespace: [ \u001b[32m\"langchain_core\"\u001b[39m, \u001b[32m\"messages\"\u001b[39m ],\n",
       "    content: \u001b[32m\"\"\u001b[39m,\n",
       "    name: \u001b[90mundefined\u001b[39m,\n",
       "    additional_kwargs: {\n",
       "      function_call: \u001b[90mundefined\u001b[39m,\n",
       "      tool_calls: [\n",
       "        {\n",
       "          id: \u001b[32m\"call_0pEdltlfSXjq20RaBFKSQOeF\"\u001b[39m,\n",
       "          type: \u001b[32m\"function\"\u001b[39m,\n",
       "          function: \u001b[36m[Object]\u001b[39m\n",
       "        }\n",
       "      ]\n",
       "    },\n",
       "    response_metadata: {\n",
       "      tokenUsage: { completionTokens: \u001b[33m33\u001b[39m, promptTokens: \u001b[33m88\u001b[39m, totalTokens: \u001b[33m121\u001b[39m },\n",
       "      finish_reason: \u001b[32m\"stop\"\u001b[39m\n",
       "    },\n",
       "    tool_calls: [\n",
       "      {\n",
       "        name: \u001b[32m\"joke\"\u001b[39m,\n",
       "        args: {\n",
       "          setup: \u001b[32m\"Why was the cat sitting on the computer?\"\u001b[39m,\n",
       "          punchline: \u001b[32m\"Because it wanted to keep an eye on the mouse!\"\u001b[39m,\n",
       "          rating: \u001b[33m7\u001b[39m\n",
       "        },\n",
       "        id: \u001b[32m\"call_0pEdltlfSXjq20RaBFKSQOeF\"\u001b[39m\n",
       "      }\n",
       "    ],\n",
       "    invalid_tool_calls: [],\n",
       "    usage_metadata: { input_tokens: \u001b[33m88\u001b[39m, output_tokens: \u001b[33m33\u001b[39m, total_tokens: \u001b[33m121\u001b[39m }\n",
       "  },\n",
       "  parsed: {\n",
       "    setup: \u001b[32m\"Why was the cat sitting on the computer?\"\u001b[39m,\n",
       "    punchline: \u001b[32m\"Because it wanted to keep an eye on the mouse!\"\u001b[39m,\n",
       "    rating: \u001b[33m7\u001b[39m\n",
       "  }\n",
       "}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const joke = z.object({\n",
    "  setup: z.string().describe(\"The setup of the joke\"),\n",
    "  punchline: z.string().describe(\"The punchline to the joke\"),\n",
    "  rating: z.number().optional().describe(\"How funny the joke is, from 1 to 10\"),\n",
    "});\n",
    "\n",
    "const structuredLlm = model.withStructuredOutput(joke, { includeRaw: true, name: \"joke\" });\n",
    "\n",
    "await structuredLlm.invoke(\"Tell me a joke about cats\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e92a98a",
   "metadata": {},
   "source": [
    "## Prompting techniques\n",
    "\n",
    "You can also prompt models to outputting information in a given format. This approach relies on designing good prompts and then parsing the output of the models. This is the only option for models that don't support `.with_structured_output()` or other built-in approaches.\n",
    "\n",
    "### Using `JsonOutputParser`\n",
    "\n",
    "The following example uses the built-in [`JsonOutputParser`](https://api.js.langchain.com/classes/langchain_core.output_parsers.JsonOutputParser.html) to parse the output of a chat model prompted to match a the given JSON schema. Note that we are adding `format_instructions` directly to the prompt from a method on the parser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6e514455",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { JsonOutputParser } from \"@langchain/core/output_parsers\";\n",
    "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
    "\n",
    "type Person = {\n",
    "    name: string;\n",
    "    height_in_meters: number;\n",
    "};\n",
    "\n",
    "type People = {\n",
    "    people: Person[];\n",
    "};\n",
    "\n",
    "const formatInstructions = `Respond only in valid JSON. The JSON object you return should match the following schema:\n",
    "{{ people: [{{ name: \"string\", height_in_meters: \"number\" }}] }}\n",
    "\n",


"Where people is an array of objects, each with a name and height_in_meters field.\n",
    "`\n",
    "\n",
    "// Set up a parser\n",
    "const parser = new JsonOutputParser<People>();\n",
    "\n",
    "// Prompt\n",
    "const prompt = await ChatPromptTemplate.fromMessages(\n",
    "    [\n",
    "        [\n",
    "            \"system\",\n",
    "            \"Answer the user query. Wrap the output in `json` tags\\n{format_instructions}\",\n",
    "        ],\n",
    "        [\n",
    "            \"human\",\n",
    "            \"{query}\",\n",
    "        ]\n",
    "    ]\n",
    ").partial({\n",
    "    format_instructions: formatInstructions,\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "082fa166",
   "metadata": {},
   "source": [
    "Let’s take a look at what information is sent to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3d73d33d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System: Answer the user query. Wrap the output in `json` tags\n",
      "Respond only in valid JSON. The JSON object you return should match the following schema:\n",
      "{{ people: [{{ name: \"string\", height_in_meters: \"number\" }}] }}\n",
      "\n",
      "Where people is an array of objects, each with a name and height_in_meters field.\n",
      "\n",
      "Human: Anna is 23 years old and she is 6 feet tall\n"
     ]
    }
   ],
   "source": [
    "const query = \"Anna is 23 years old and she is 6 feet tall\"\n",
    "\n",
    "console.log((await prompt.format({ query })).toString())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081956b9",
   "metadata": {},
   "source": [
    "And now let's invoke it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8d6b3d17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{ people: [ { name: \u001b[32m\"Anna\"\u001b[39m, height_in_meters: \u001b[33m1.83\u001b[39m } ] }"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "const chain = prompt.pipe(model).pipe(parser);\n",
    "\n",
    "await chain.invoke({ query })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6732dd87",
   "metadata": {},
   "source": [
    "For a deeper dive into using output parsers with prompting techniques for structured output, see [this guide](/docs/how_to/output_parser_structured).\n",
    "\n",
    "### Custom Parsing\n",
    "\n",
    "You can also create a custom prompt and parser with [LangChain Expression Language (LCEL)](/docs/concepts/#langchain-expression-language), using a plain function to parse the output from the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "525721b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { AIMessage } from \"@langchain/core/messages\";\n",
    "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
    "\n",
    "type Person = {\n",
    "    name: string;\n",
    "    height_in_meters: number;\n",
    "};\n",
    "\n",
    "type People = {\n",
    "    people: Person[];\n",
    "};\n",
    "\n",
    "const schema = `{{ people: [{{ name: \"string\", height_in_meters: \"number\" }}] }}`\n",
    "\n",
    "// Prompt\n",
    "const prompt = await ChatPromptTemplate.fromMessages(\n",
    "    [\n",
    "        [\n",
    "            \"system\",\n",
    "            `Answer the user query. Output your answer as JSON that\n",
    "matches the given schema: \\`\\`\\`json\\n{schema}\\n\\`\\`\\`.\n",
    "Make sure to wrap the answer in \\`\\`\\`json and \\`\\`\\` tags`\n",
    "        ],\n",
    "        [\n",
    "            \"human\",\n",
    "            \"{query}\",\n",
    "        ]\n",
    "    ]\n",
    ").partial({\n",
    "    schema\n",
    "});\n",
    "\n",
    "/**\n",
    " * Custom extractor\n",
    " * \n",
    " * Extracts JSON content from a string where\n",
    " * JSON is embedded between ```json and ``` tags.\n",
    " */\n",
    "const extractJson = (output: AIMessage): Array<People> => {\n",
    "    const text = output.content as string;\n",
    "    // Define the regular expression pattern to match JSON blocks\n",
    "    const pattern = /```json(.*?)```/gs;\n",
    "\n",
    "    // Find all non-overlapping matches of the pattern in the string\n",
    "    const matches = text.match(pattern);\n",
    "\n",
    "    // Process each match, attempting to parse it as JSON\n",
    "    try {\n",
    "        return matches?.map(match => {\n",
    "            // Remove the markdown code block syntax to isolate the JSON string\n",
    "            const jsonStr = match.replace(/```json|```/g, '').trim();\n",
    "            return JSON.parse(jsonStr);\n",
    "        }) ?? [];\n",
    "    } catch (error) {\n",
    "        throw new Error(`Failed to parse: ${output}`);\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f1bc8f7",
   "metadata": {},
   "source": [
    "Here is the prompt sent to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c8a30d0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System: Answer the user query. Output your answer as JSON that\n",
      "matches the given schema: ```json\n",
      "{{ people: [{{ name: \"string\", height_in_meters: \"number\" }}] }}\n",
      "```.\n",
      "Make sure to wrap the answer in ```json and ``` tags\n",
      "Human: Anna is 23 years old and she is 6 feet tall\n"
     ]
    }
   ],
   "source": [
    "const query = \"Anna is 23 years old and she is 6 feet tall\"\n",
    "\n",
    "console.log((await prompt.format({ query })).toString())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec018893",
   "metadata": {},
   "source": [
    "And here's what it looks like when we invoke it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e1e7baf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\n",
       "  { people: [ { name: \u001b[32m\"Anna\"\u001b[39m, height_in_meters: \u001b[33m1.83\u001b[39m } ] }\n",
       "]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import { RunnableLambda } from \"@langchain/core/runnables\";\n",
    "\n",
    "const chain = prompt.pipe(model).pipe(new RunnableLambda({ func: extractJson }));\n",
    "\n",
    "await chain.invoke({ query })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a39221a",
   "metadata": {},
   "source": [


# How to load JSON data

> [JSON (JavaScript Object Notation)](https://en.wikipedia.org/wiki/JSON) is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays (or other serializable values).

> [JSON Lines](https://jsonlines.org/) is a file format where each line is a valid JSON value.

The JSON loader uses [JSON pointer](https://github.com/janl/node-jsonpointer) to target keys in your JSON files you want to target.

### No JSON pointer example

The most simple way of using it is to specify no JSON pointer.
The loader will load all strings it finds in the JSON object.

Example JSON file:

```json
{
  "texts": ["This is a sentence.", "This is another sentence."]
}
```

Example code:

```typescript
import { JSONLoader } from "langchain/document_loaders/fs/json";

const loader = new JSONLoader("src/document_loaders/example_data/example.json");

const docs = await loader.load();
/*
[
  Document {
    "metadata": {
      "blobType": "application/json",
      "line": 1,
      "source": "blob",
    },
    "pageContent": "This is a sentence.",
  },
  Document {
    "metadata": {
      "blobType": "application/json",
      "line": 2,
      "source": "blob",
    },
    "pageContent": "This is another sentence.",
  },
]
*/
```

### Using JSON pointer example

You can do a more advanced scenario by choosing which keys in your JSON object you want to extract string from.

In this example, we want to only extract information from "from" and "surname" entries.

```json
{
  "1": {
    "body": "BD 2023 SUMMER",
    "from": "LinkedIn Job",
    "labels": ["IMPORTANT", "CATEGORY_UPDATES", "INBOX"]
  },
  "2": {
    "body": "Intern, Treasury and other roles are available",
    "from": "LinkedIn Job2",
    "labels": ["IMPORTANT"],
    "other": {
      "name": "plop",
      "surname": "bob"
    }
  }
}
```

Example code:

```typescript
import { JSONLoader } from "langchain/document_loaders/fs/json";

const loader = new JSONLoader(
  "src/document_loaders/example_data/example.json",
  ["/from", "/surname"]
);

const docs = await loader.load();
/*
[
  Document {
    pageContent: 'LinkedIn Job',
    metadata: { source: './src/json/example.json', line: 1 }
  },
  Document {
    pageContent: 'LinkedIn Job2',
    metadata: { source: './src/json/example.json', line: 2 }
  },
  Document {
    pageContent: 'bob',
    metadata: { source: './src/json/example.json', line: 3 }
  }
]
**/
```


---
sidebar_position: 5
---

# How to track token usage

:::info Prerequisites

This guide assumes familiarity with the following concepts:

- [Chat models](/docs/concepts/#chat-models)

:::

This notebook goes over how to track your token usage for specific calls.

## Using `AIMessage.usage_metadata`

A number of model providers return token usage information as part of the chat generation response. When available, this information will be included on the `AIMessage` objects produced by the corresponding model.

LangChain `AIMessage` objects include a [`usage_metadata`](https://api.js.langchain.com/classes/langchain_core.messages.AIMessage.html#usage_metadata) attribute for supported providers. When populated, this attribute will be an object with standard keys (e.g., "input_tokens" and "output_tokens").

#### OpenAI

import CodeBlock from "@theme/CodeBlock";
import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm install @langchain/openai @langchain/core
```

import UsageMetadataExample from "@examples/models/chat/usage_metadata.ts";

<CodeBlock language="typescript">{UsageMetadataExample}</CodeBlock>

#### Anthropic

```bash npm2yarn
npm install @langchain/anthropic @langchain/core
```

import UsageMetadataExampleAnthropic from "@examples/models/chat/usage_metadata_anthropic.ts";

<CodeBlock language="typescript">{UsageMetadataExampleAnthropic}</CodeBlock>

## Using `AIMessage.response_metadata`

A number of model providers return token usage information as part of the chat generation response. When available, this is included in the `AIMessage.response_metadata` field.

#### OpenAI

import Example from "@examples/models/chat/token_usage_tracking.ts";

<CodeBlock language="typescript">{Example}</CodeBlock>

#### Anthropic

import AnthropicExample from "@examples/models/chat/token_usage_tracking_anthropic.ts";

<CodeBlock language="typescript">{AnthropicExample}</CodeBlock>

## Streaming

Some providers support token count metadata in a streaming context.

#### OpenAI

For example, OpenAI will return a message chunk at the end of a stream with token usage information. This behavior is supported by `@langchain/openai` >= 0.1.0 and can be enabled by passing a `stream_options` parameter when making your call.

:::info
By default, the last message chunk in a stream will include a `finish_reason` in the message's `response_metadata` attribute. If we include token usage in streaming mode, an additional chunk containing usage metadata will be added to the end of the stream, such that `finish_reason` appears on the second to last message chunk.
:::

import OpenAIStreamTokens from "@examples/models/chat/integration_openai_stream_tokens.ts";

<CodeBlock language="typescript">{OpenAIStreamTokens}</CodeBlock>

## Using callbacks

You can also use the `handleLLMEnd` callback to get the full output from the LLM, including token usage for supported models.
Here's an example of how you could do that:

import CallbackExample from "@examples/models/chat/token_usage_tracking_callback.ts";

<CodeBlock language="typescript">{CallbackExample}</CodeBlock>

## Next steps

You've now seen a few examples of how to track chat model token usage for supported providers.

Next, check out the other how-to guides on chat models in this section, like [how to get a model to return structured output](/docs/how_to/structured_output) or [how to add caching to your chat models](/docs/how_to/chat_model_caching).


# How to do retrieval with contextual compression

:::info Prerequisites

This guide assumes familiarity with the following concepts:

- [Retrievers](/docs/concepts/#retrievers)
- [Retrieval-augmented generation (RAG)](/docs/tutorials/rag)

:::

One challenge with retrieval is that usually you don't know the specific queries your document storage system will face when you ingest data into the system. This means that the information most relevant to a query may be buried in a document with a lot of irrelevant text. Passing that full document through your application can lead to more expensive LLM calls and poorer responses.

Contextual compression is meant to fix this. The idea is simple: instead of immediately returning retrieved documents as-is, you can compress them using the context of the given query, so that only the relevant information is returned. “Compressing” here refers to both compressing the contents of an individual document and filtering out documents wholesale.

To use the Contextual Compression Retriever, you'll need:

- a base retriever
- a Document Compressor

The Contextual Compression Retriever passes queries to the base retriever, takes the initial documents and passes them through the Document Compressor. The Document Compressor takes a list of documents and shortens it by reducing the contents of documents or dropping documents altogether.

## Using a vanilla vector store retriever

Let's start by initializing a simple vector store retriever and storing the 2023 State of the Union speech (in chunks).
Given an example question, our retriever returns one or two relevant docs and a few irrelevant docs, and even the relevant docs have a lot of irrelevant information in them.
To extract all the context we can, we use an `LLMChainExtractor`, which will iterate over the initially returned documents and extract from each only the content that is relevant to the query.

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm install @langchain/openai @langchain/community @langchain/core
```

import CodeBlock from "@theme/CodeBlock";
import Example from "@examples/retrievers/contextual_compression.ts";

<CodeBlock language="typescript">{Example}</CodeBlock>

## `EmbeddingsFilter`

Making an extra LLM call over each retrieved document is expensive and slow. The `EmbeddingsFilter` provides a cheaper and faster option by embedding the documents and query and only returning those documents which have sufficiently similar embeddings to the query.

This is most useful for non-vector store retrievers where we may not have control over the returned chunk size, or as part of a pipeline, outlined below.

Here's an example:

import EmbeddingsFilterExample from "@examples/retrievers/embeddings_filter.ts";

<CodeBlock language="typescript">{EmbeddingsFilterExample}</CodeBlock>

## Stringing compressors and document transformers together

Using the `DocumentCompressorPipeline` we can also easily combine multiple compressors in sequence. Along with compressors we can add BaseDocumentTransformers to our pipeline, which don't perform any contextual compression but simply perform some transformation on a set of documents.
For example `TextSplitters` can be used as document transformers to split documents into smaller pieces, and the `EmbeddingsFilter` can be used to filter out documents based on similarity of the individual chunks to the input query.

Below we create a compressor pipeline by first splitting raw webpage documents retrieved from the [Tavily web search API retriever](/docs/integrations/retrievers/tavily) into smaller chunks, then filtering based on relevance to the query.
The result is smaller chunks that are semantically similar to the input query.
This skips the need to add documents to a vector store to perform similarity search, which can be useful for one-off use cases:

import DocumentCompressorPipelineExample from "@examples/retrievers/document_compressor_pipeline.ts";

<CodeBlock language="typescript">{DocumentCompressorPipelineExample}</CodeBlock>

## Next steps

You've now learned a few ways to use contextual compression to remove bad data from your results.

See the individual sections for deeper dives on specific retrievers, the [broader tutorial on RAG](/docs/tutorials/rag), or this section to learn how to
[create your own custom retriever over any data source](/docs/how_to/custom_retriever/).


# How to load data from a directory

This covers how to load all documents in a directory.

The second argument is a map of file extensions to loader factories. Each file will be passed to the matching loader, and the resulting documents will be concatenated together.

Example folder:

```text
src/document_loaders/example_data/example/
├── example.json
├── example.jsonl
├── example.txt
└── example.csv
```

Example code:

```typescript
import { DirectoryLoader } from "langchain/document_loaders/fs/directory";
import {
  JSONLoader,
  JSONLinesLoader,
} from "langchain/document_loaders/fs/json";
import { TextLoader } from "langchain/document_loaders/fs/text";
import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";

const loader = new DirectoryLoader(
  "src/document_loaders/example_data/example",
  {
    ".json": (path) => new JSONLoader(path, "/texts"),
    ".jsonl": (path) => new JSONLinesLoader(path, "/html"),
    ".txt": (path) => new TextLoader(path),
    ".csv": (path) => new CSVLoader(path, "text"),
  }
);
const docs = await loader.load();
console.log({ docs });
```


---
sidebar_position: 5
keywords: [RunnablePassthrough, LCEL]
---

# How to pass through arguments from one step to the next

This guide assumes familiarity with the following concepts:

- [LangChain Expression Language (LCEL)](/docs/concepts/#langchain-expression-language)

- [Calling runnables in parallel](/docs/how_to/parallel/)

When composing chains with several steps, sometimes you will want to pass data from previous steps unchanged for use as input to a later step. The [`RunnablePassthrough`](https://api.js.langchain.com/classes/langchain_core.runnables.RunnablePassthrough.html) class allows you to do just this, and is typically is used in conjuction with a [RunnableParallel](/docs/how_to/parallel/) to pass data through to a later step in your constructed chains.

import { RunnableParallel, RunnablePassthrough } from "@langchain/core/runnables";

const runnable = RunnableParallel.from({

passed: new RunnablePassthrough<{ num: number }>(),

modified: (input: { num: number }) => input.num + 1,

As seen above, `passed` key was called with `RunnablePassthrough()` and so it simply passed on `{'num': 1}`.

We also set a second key in the map with `modified`. This uses a lambda to set a single value adding 1 to the num, which resulted in `modified` key with the value of `2`.

In the example below, we see a more real-world use case where we use `RunnablePassthrough` along with `RunnableParallel` in a chain to properly format inputs to a prompt:

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

import { StringOutputParser } from "@langchain/core/output_parsers";

import { ChatPromptTemplate } from "@langchain/core/prompts";

import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";

import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";

import { MemoryVectorStore } from "langchain/vectorstores/memory";

const vectorstore = await MemoryVectorStore.fromDocuments([

{ pageContent: "harrison worked at kensho", metadata: {} }

const retriever = vectorstore.asRetriever();

const template = `Answer the question based only on the following context:

const prompt = ChatPromptTemplate.fromTemplate(template);

const model = new ChatOpenAI({ model: "gpt-4o" });

const retrievalChain = RunnableSequence.from([

context: retriever.pipe((docs) => docs[0].pageContent),

await retrievalChain.invoke("where did harrison work?");

Here the input to prompt is expected to be a map with keys `"context"` and `"question"`. The user input is just the question. So we need to get the context using our retriever and passthrough the user input under the `"question"` key. The `RunnablePassthrough` allows us to pass on the user's question to the prompt and model.

Now you've learned how to pass data through your chains to help to help format the data flowing through your chains.

To learn more, see the other how-to guides on runnables in this section.


This guide assumes familiarity with the following concepts:

- [How to use a model to call tools](/docs/how_to/tool_calling)

In order to force our LLM to select a specific tool, we can use the `tool_choice` parameter to ensure certain behavior. First, let's define our model and tools:

import { tool } from '@langchain/core/tools';

import { ChatOpenAI } from '@langchain/openai';

For example, we can force our tool to call the multiply tool by using the following code:

"name": "Multiply",

"args": {

"a": 2,

"b": 4

"type": "tool_call",

"id": "call_d5isFbUkn17Wjr6yEtNz7dDF"

const llmForcedToMultiply = llm.bindTools(tools, {

const multiplyResult = await llmForcedToMultiply.invoke("what is 2 + 4");

Even if we pass it something that doesn't require multiplcation - it will still call the tool!

We can also just force our tool to select at least one of our tools by passing `"any"` (or for OpenAI models, the equivalent, `"required"`) to the `tool_choice` parameter.

"name": "add",

"args": {

"a": 2,

"b": 3

"type": "tool_call",

"id": "call_La72g7Aj0XHG0pfPX6Dwg2vT"

const llmForcedToUseTool = llm.bindTools(tools, {

const anyToolResult = await llmForcedToUseTool.invoke("What day is today?");


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "94c3ad61",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 3\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b91e03f1",
   "metadata": {},
   "source": [
    "# How to use few shot examples\n",
    "\n",
    "In this guide, we'll learn how to create a simple prompt template that provides the model with example inputs and outputs when generating. Providing the LLM with a few such examples is called few-shotting, and is a simple yet powerful way to guide generation and in some cases drastically improve model performance.\n",
    "\n",
    "A few-shot prompt template can be constructed from either a set of examples, or from an [Example Selector](https://api.js.langchain.com/classes/langchain_core.example_selectors.BaseExampleSelector.html) class responsible for choosing a subset of examples from the defined set.\n",
    "\n",
    "This guide will cover few-shotting with string prompt templates. For a guide on few-shotting with chat messages for chat models, see [here](/docs/how_to/few_shot_examples_chat/).\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [Prompt templates](/docs/concepts/#prompt-templates)\n",
    "- [Example selectors](/docs/concepts/#example-selectors)\n",
    "- [LLMs](/docs/concepts/#llms)\n",
    "- [Vectorstores](/docs/concepts/#vectorstores)\n",
    "\n",
    ":::\n",
    "\n",
    "## Create a formatter for the few-shot examples\n",
    "\n",
    "Configure a formatter that will format the few-shot examples into a string. This formatter should be a `PromptTemplate` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4e70bce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import { PromptTemplate } from \"@langchain/core/prompts\";\n",
    "\n",
    "const examplePrompt = PromptTemplate.fromTemplate(\"Question: {question}\\n{answer}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50846ad4",
   "metadata": {},
   "source": [
    "## Creating the example set\n",
    "\n",
    "Next, we'll create a list of few-shot examples. Each example should be a dictionary representing an example input to the formatter prompt we defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a44be840",
   "metadata": {},
   "outputs": [],
   "source": [
    "const examples = [\n",
    "    {\n",
    "      question: \"Who lived longer, Muhammad Ali or Alan Turing?\",\n",
    "      answer: `\n",
    "  Are follow up questions needed here: Yes.\n",
    "  Follow up: How old was Muhammad Ali when he died?\n",
    "  Intermediate answer: Muhammad Ali was 74 years old when he died.\n",
    "  Follow up: How old was Alan Turing when he died?\n",
    "  Intermediate answer: Alan Turing was 41 years old when he died.\n",
    "  So the final answer is: Muhammad Ali\n",
    "  `\n",
    "    },\n",
    "    {\n",
    "      question: \"When was the founder of craigslist born?\",\n",
    "      answer: `\n",
    "  Are follow up questions needed here: Yes.\n",
    "  Follow up: Who was the founder of craigslist?\n",
    "  Intermediate answer: Craigslist was founded by Craig Newmark.\n",
    "  Follow up: When was Craig Newmark born?\n",
    "  Intermediate answer: Craig Newmark was born on December 6, 1952.\n",
    "  So the final answer is: December 6, 1952\n",
    "  `\n",
    "    },\n",
    "    {\n",
    "      question: \"Who was the maternal grandfather of George Washington?\",\n",
    "      answer: `\n",
    "  Are follow up questions needed here: Yes.\n",
    "  Follow up: Who was the mother of George Washington?\n",
    "  Intermediate answer: The mother of George Washington was Mary Ball Washington.\n",
    "  Follow up: Who was the father of Mary Ball Washington?\n",
    "  Intermediate answer: The father of Mary Ball Washington was Joseph Ball.\n",
    "  So the final answer is: Joseph Ball\n",
    "  `\n",
    "    },\n",
    "    {\n",
    "      question: \"Are both the directors of Jaws and Casino Royale from the same country?\",\n",
    "      answer: `\n",
    "  Are follow up questions needed here: Yes.\n",
    "  Follow up: Who is the director of Jaws?\n",
    "  Intermediate Answer: The director of Jaws is Steven Spielberg.\n",
    "  Follow up: Where is Steven Spielberg from?\n",
    "  Intermediate Answer: The United States.\n",
    "  Follow up: Who is the director of Casino Royale?\n",
    "  Intermediate Answer: The director of Casino Royale is Martin Campbell.\n",
    "  Follow up: Where is Martin Campbell from?\n",
    "  Intermediate Answer: New Zealand.\n",
    "  So the final answer is: No\n",
    "  `\n",
    "    }\n",
    "  ];"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dad66af1",
   "metadata": {},
   "source": [
    "### Pass the examples and formatter to `FewShotPromptTemplate`\n",
    "\n",
    "Finally, create a [`FewShotPromptTemplate`](https://api.js.langchain.com/classes/langchain_core.prompts.FewShotPromptTemplate.html) object. This object takes in the few-shot examples and the formatter for the few-shot examples. When this `FewShotPromptTemplate` is formatted, it formats the passed examples using the `examplePrompt`, then and adds them to the final prompt before `suffix`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e76fa1ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Question: Who lived longer, Muhammad Ali or Alan Turing?\n",
      "\n",
      "  Are follow up questions needed here: Yes.\n",
      "  Follow up: How old was Muhammad Ali when he died?\n",
      "  Intermediate answer: Muhammad Ali was 74 years old when he died.\n",
      "  Follow up: How old was Alan Turing when he died?\n",
      "  Intermediate answer: Alan Turing was 41 years old when he died.\n",
      "  So the final answer is: Muhammad Ali\n",
      "  \n",
      "\n",
      "Question: When was the founder of craigslist born?\n",
      "\n",
      "  Are follow up questions needed here: Yes.\n",
      "  Follow up: Who was the founder of craigslist?\n",
      "  Intermediate answer: Craigslist was founded by Craig Newmark.\n",
      "  Follow up: When was Craig Newmark born?\n",
      "  Intermediate answer: Craig Newmark was born on December 6, 1952.\n",
      "  So the final answer is: December 6, 1952\n",
      "  \n",
      "\n",
      "Question: Who was the maternal grandfather of George Washington?\n",
      "\n",
      "  Are follow up questions needed here: Yes.\n",
      "  Follow up: Who was the mother of George Washington?\n",
      "  Intermediate answer: The mother of George Washington was Mary Ball Washington.\n",
      "  Follow up: Who was the father of Mary Ball Washington?\n",
      "  Intermediate answer: The father of Mary Ball Washington was Joseph Ball.\n",
      "  So the final answer is: Joseph Ball\n",
      "  \n",
      "\n",
      "Question: Are both the directors of Jaws and Casino Royale from the same country?\n",
      "\n",
      "  Are follow up questions needed here: Yes.\n",
      "  Follow up: Who is the director of Jaws?\n",
      "  Intermediate Answer: The director of Jaws is Steven Spielberg.\n",
      "  Follow up: Where is Steven Spielberg from?\n",
      "  Intermediate Answer: The United States.\n",
      "  Follow up: Who is the director of Casino Royale?\n",
      "  Intermediate Answer: The director of Casino Royale is Martin Campbell.\n",


keywords: [RunnablePassthrough, assign, LCEL]

- [LangChain Expression Language (LCEL)](/docs/concepts/#langchain-expression-language)

- [Calling runnables in parallel](/docs/how_to/parallel/)

- [Passing data through](/docs/how_to/passthrough)

An alternate way of [passing data through](/docs/how_to/passthrough) steps of a chain is to leave the current values of the chain state unchanged while assigning a new value under a given key. The [`RunnablePassthrough.assign()`](https://api.js.langchain.com/classes/langchain_core.runnables.RunnablePassthrough.html#assign-2) static method takes an input value and adds the extra arguments passed to the assign function.

This is useful in the common [LangChain Expression Language](/docs/concepts/#langchain-expression-language) pattern of additively creating a dictionary to use as input to a later step.

{ extra: { num: 1, mult: 3, modified: 2 } }

import { RunnableParallel, RunnablePassthrough } from "@langchain/core/runnables";

const runnable = RunnableParallel.from({

mult: (input: { num: number }) => input.num * 3,

modified: (input: { num: number }) => input.num + 1

- The input to the chain is `{"num": 1}`. This is passed into a `RunnableParallel`, which invokes the runnables it is passed in parallel with that input.

- The value under the `extra` key is invoked. `RunnablePassthrough.assign()` keeps the original keys in the input dict (`{"num": 1}`), and assigns a new key called `mult`. The value is `lambda x: x["num"] * 3)`, which is `3`. Thus, the result is `{"num": 1, "mult": 3}`.

- `{"num": 1, "mult": 3}` is returned to the `RunnableParallel` call, and is set as the value to the key `extra`.

- At the same time, the `modified` key is called. The result is `2`, since the lambda extracts a key called `"num"` from its input and adds one.

Thus, the result is `{'extra': {'num': 1, 'mult': 3}, 'modified': 2}`.

One convenient feature of this method is that it allows values to pass through as soon as they are available. To show this off, we'll use `RunnablePassthrough.assign()` to immediately return source docs in a retrieval chain:

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

{ question: "where did harrison work?" }

{ context: "harrison worked at kensho" }

{ output: "" }

{ output: "H" }

{ output: "arrison" }

{ output: " worked" }

{ output: " at" }

{ output: " Kens" }

{ output: "ho" }

{ output: "." }

{ output: "" }

import { StringOutputParser } from "@langchain/core/output_parsers";

import { ChatPromptTemplate } from "@langchain/core/prompts";

import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";

import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";

import { MemoryVectorStore } from "langchain/vectorstores/memory";

const vectorstore = await MemoryVectorStore.fromDocuments([

{ pageContent: "harrison worked at kensho", metadata: {} }

const retriever = vectorstore.asRetriever();

const template = `Answer the question based only on the following context:

const prompt = ChatPromptTemplate.fromTemplate(template);

const model = new ChatOpenAI({ model: "gpt-4o" });

const generationChain = prompt.pipe(model).pipe(new StringOutputParser());

const retrievalChain = RunnableSequence.from([

context: retriever.pipe((docs) => docs[0].pageContent),

RunnablePassthrough.assign({ output: generationChain }),

const stream = await retrievalChain.stream("where did harrison work?");

We can see that the first chunk contains the original `"question"` since that is immediately available. The second chunk contains `"context"` since the retriever finishes second. Finally, the output from the `generation_chain` streams in chunks as soon as it is available.

Now you've learned how to pass data through your chains to help to help format the data flowing through your chains.

To learn more, see the other how-to guides on runnables in this section.


# How use a vector store to retrieve data

:::info Prerequisites

This guide assumes familiarity with the following concepts:

- [Vector stores](/docs/concepts/#vectorstores)
- [Retrievers](/docs/concepts/#retrievers)
- [Text splitters](/docs/concepts#text-splitters)
- [Chaining runnables](/docs/how_to/sequence/)

:::

Vector stores can be converted into retrievers using the [`.asRetriever()`](https://api.js.langchain.com/classes/langchain_core.vectorstores.VectorStore.html#asRetriever) method, which allows you to more easily compose them in chains.

Below, we show a retrieval-augmented generation (RAG) chain that performs question answering over documents using the following steps:

1. Initialize an vector store
2. Create a retriever from that vector store
3. Compose a question answering chain
4. Ask questions!

Each of the steps has multiple sub steps and potential configurations, but we'll go through one common flow.
First, install the required dependency:

import CodeBlock from "@theme/CodeBlock";

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm install @langchain/openai @langchain/core
```

You can download the `state_of_the_union.txt` file [here](https://github.com/langchain-ai/langchain/blob/master/docs/docs/modules/state_of_the_union.txt).

import RetrievalQAExample from "@examples/chains/retrieval_qa.ts";

<CodeBlock language="typescript">{RetrievalQAExample}</CodeBlock>

Let's walk through what's happening here.

1. We first load a long text and split it into smaller documents using a text splitter.
   We then load those documents (which also embeds the documents using the passed `OpenAIEmbeddings` instance) into HNSWLib, our vector store, creating our index.

2. Though we can query the vector store directly, we convert the vector store into a retriever to return retrieved documents in the right format for the question answering chain.

3. We initialize a retrieval chain, which we'll call later in step 4.

4. We ask questions!

## Next steps

You've now learned how to convert a vector store as a retriever.

See the individual sections for deeper dives on specific retrievers, the [broader tutorial on RAG](/docs/tutorials/rag), or this section to learn how to
[create your own custom retriever over any data source](/docs/how_to/custom_retriever/).


"Message chunks are additive by design – one can simply add them up using the `.concat()` method to get the state of the response so far!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AIMessageChunk {\n",
      "  lc_serializable: true,\n",
      "  lc_kwargs: {\n",
      "    content: \"Hello! I'm a\",\n",
      "    additional_kwargs: {},\n",
      "    response_metadata: { prompt: 0, completion: 0, finish_reason: null },\n",
      "    tool_call_chunks: [],\n",
      "    id: 'chatcmpl-9lO8YUEcX7rqaxxevelHBtl1GaWoo',\n",
      "    tool_calls: [],\n",
      "    invalid_tool_calls: []\n",
      "  },\n",
      "  lc_namespace: [ 'langchain_core', 'messages' ],\n",
      "  content: \"Hello! I'm a\",\n",
      "  name: undefined,\n",
      "  additional_kwargs: {},\n",
      "  response_metadata: { prompt: 0, completion: 0, finish_reason: null },\n",
      "  id: 'chatcmpl-9lO8YUEcX7rqaxxevelHBtl1GaWoo',\n",
      "  tool_calls: [],\n",
      "  invalid_tool_calls: [],\n",
      "  tool_call_chunks: [],\n",
      "  usage_metadata: undefined\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "let finalChunk = chunks[0];\n",
    "\n",
    "for (const chunk of chunks.slice(1, 5)) {\n",
    "  finalChunk = finalChunk.concat(chunk);\n",
    "}\n",
    "\n",
    "finalChunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chains\n",
    "\n",
    "Virtually all LLM applications involve more steps than just a call to a language model.\n",
    "\n",
    "Let’s build a simple chain using `LangChain Expression Language` (`LCEL`) that combines a prompt, model and a parser and verify that streaming works.\n",
    "\n",
    "We will use `StringOutputParser` to parse the output from the model. This is a simple parser that extracts the content field from an `AIMessageChunk`, giving us the `token` returned by the model.\n",
    "\n",
    ":::{.callout-tip}\n",
    "LCEL is a declarative way to specify a “program” by chainining together different LangChain primitives. Chains created using LCEL benefit from an automatic implementation of stream, allowing streaming of the final output. In fact, chains created with LCEL implement the entire standard Runnable interface.\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|\n",
      "Sure|\n",
      ",|\n",
      " here's|\n",
      " a|\n",
      " joke|\n",
      " for|\n",
      " you|\n",
      ":\n",
      "\n",
      "|\n",
      "Why|\n",
      " did|\n",
      " the|\n",
      " par|\n",
      "rot|\n",
      " sit|\n",
      " on|\n",
      " the|\n",
      " stick|\n",
      "?\n",
      "\n",
      "|\n",
      "Because|\n",
      " it|\n",
      " wanted|\n",
      " to|\n",
      " be|\n",
      " a|\n",
      " \"|\n",
      "pol|\n",
      "ly|\n",
      "-stick|\n",
      "-al|\n",
      "\"|\n",
      " observer|\n",
      "!|\n",
      "|\n",
      "|\n"
     ]
    }
   ],
   "source": [
    "import { StringOutputParser } from \"@langchain/core/output_parsers\";\n",
    "import { ChatPromptTemplate } from \"@langchain/core/prompts\";\n",
    "\n",
    "const prompt = ChatPromptTemplate.fromTemplate(\"Tell me a joke about {topic}\");\n",
    "\n",
    "const parser = new StringOutputParser();\n",
    "\n",
    "const chain = prompt.pipe(model).pipe(parser);\n",
    "\n",
    "const stream = await chain.stream({\n",
    "  topic: \"parrot\",\n",
    "});\n",
    "\n",
    "for await (const chunk of stream) {\n",
    "  console.log(`${chunk}|`)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ":::{.callout-note}\n",
    "You do not have to use the `LangChain Expression Language` to use LangChain and can instead rely on a standard **imperative** programming approach by\n",
    "caling `invoke`, `batch` or `stream` on each component individually, assigning the results to variables and then using them downstream as you see fit.\n",
    "\n",
    "If that works for your needs, then that's fine by us 👌!\n",
    ":::\n",
    "\n",
    "### Working with Input Streams\n",
    "\n",
    "What if you wanted to stream JSON from the output as it was being generated?\n",
    "\n",
    "If you were to rely on `JSON.parse` to parse the partial json, the parsing would fail as the partial json wouldn't be valid json.\n",
    "\n",
    "You'd likely be at a complete loss of what to do and claim that it wasn't possible to stream JSON.\n",
    "\n",
    "Well, turns out there is a way to do it - the parser needs to operate on the **input stream**, and attempt to \"auto-complete\" the partial json into a valid state.\n",
    "\n",
    "Let's see such a parser in action to understand what this means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  countries: [\n",
      "    { name: 'France', population: 67390000 },\n",
      "    { name: 'Spain', population: 47350000 },\n",
      "    { name: 'Japan', population: 125800000 }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import { JsonOutputParser } from \"@langchain/core/output_parsers\"\n",
    "\n",
    "const chain = model.pipe(new JsonOutputParser());\n",
    "const stream = await chain.stream(\n",
    "  `Output a list of the countries france, spain and japan and their populations in JSON format. Use a dict with an outer key of \"countries\" which contains a list of countries. Each country should have the key \"name\" and \"population\"`\n",
    ");\n",
    "\n",
    "for await (const chunk of stream) {\n",
    "  console.log(chunk);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's **break** streaming. We'll use the previous example and append an extraction function at the end that extracts the country names from the finalized JSON. Since this new last step is just a function call with no defined streaming behavior, the streaming output from previous steps is aggregated, then passed as a single input to the function.\n",
    "\n",
    ":::{.callout-warning}\n",
    "Any steps in the chain that operate on **finalized inputs** rather than on **input streams** can break streaming functionality via `stream`.\n",
    ":::\n",
    "\n",
    ":::{.callout-tip}\n",
    "Later, we will discuss the `streamEvents` API which streams results from intermediate steps. This API will stream results from intermediate steps even if the chain contains steps that only operate on **finalized inputs**.\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"France\",\"Spain\",\"Japan\"]\n"
     ]
    }
   ],
   "source": [


{
  "cells": [
    {
      "cell_type": "raw",
      "id": "0e77c293-4049-43be-ba49-ff9daeefeee7",
      "metadata": {},
      "source": [
        "---\n",
        "sidebar_position: 4\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "14d3fd06",
      "metadata": {},
      "source": [
        "# How to do per-user retrieval\n",
        "\n",
        ":::info Prerequisites\n",
        "\n",
        "This guide assumes familiarity with the following:\n",
        "\n",
        "- [Retrieval-augmented generation](/docs/tutorials/rag/)\n",
        "\n",
        ":::\n",
        "\n",
        "When building a retrieval app, you often have to build it with multiple users in\n",
        "mind. This means that you may be storing data not just for one user, but for\n",
        "many different users, and they should not be able to see each other's data. This\n",
        "means that you need to be able to configure your retrieval chain to only\n",
        "retrieve certain information. This generally involves two steps.\n",
        "\n",
        "**Step 1: Make sure the retriever you are using supports multiple users**\n",
        "\n",
        "At the moment, there is no unified flag or filter for this in LangChain. Rather,\n",
        "each vectorstore and retriever may have their own, and may be called different\n",
        "things (namespaces, multi-tenancy, etc). For vectorstores, this is generally\n",
        "exposed as a keyword argument that is passed in during `similaritySearch`. By\n",
        "reading the documentation or source code, figure out whether the retriever you\n",
        "are using supports multiple users, and, if so, how to use it.\n",
        "\n",
        "**Step 2: Add that parameter as a configurable field for the chain**\n",
        "\n",
        "The LangChain `config` object is passed through to every Runnable. Here you can\n",
        "add any fields you'd like to the `configurable` object. Later, inside the chain\n",
        "we can extract these fields.\n",
        "\n",
        "**Step 3: Call the chain with that configurable field**\n",
        "\n",
        "Now, at runtime you can call this chain with configurable field.\n",
        "\n",
        "## Code Example\n",
        "\n",
        "Let's see a concrete example of what this looks like in code. We will use\n",
        "Pinecone for this example."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c8ccbef7",
      "metadata": {},
      "source": [
        "## Setup\n",
        "\n",
        "### Install dependencies\n",
        "\n",
        "```{=mdx}\n",
        "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n",
        "import Npm2Yarn from \"@theme/Npm2Yarn\";\n",
        "\n",
        "<IntegrationInstallTooltip></IntegrationInstallTooltip>\n",
        "\n",
        "<Npm2Yarn>\n",
        "  @langchain/pinecone @langchain/openai @langchain/core @pinecone-database/pinecone\n",
        "</Npm2Yarn>\n",
        "```\n",
        "\n",
        "### Set environment variables\n",
        "\n",
        "We'll use OpenAI and Pinecone in this example:\n",
        "\n",
        "```env\n",
        "OPENAI_API_KEY=your-api-key\n",
        "\n",
        "PINECONE_API_KEY=your-api-key\n",
        "PINECONE_INDEX=your-index-name\n",
        "\n",
        "# Optional, use LangSmith for best-in-class observability\n",
        "LANGSMITH_API_KEY=your-api-key\n",
        "LANGCHAIN_TRACING_V2=true\n",
        "\n",
        "# Reduce tracing latency if you are not in a serverless environment\n",
        "# LANGCHAIN_CALLBACKS_BACKGROUND=true\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "7345de3c",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[ \u001b[32m\"77b8f174-9d89-4c6c-b2ab-607fe3913b2d\"\u001b[39m ]"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
        "import { PineconeStore } from \"@langchain/pinecone\";\n",
        "import { Pinecone } from \"@pinecone-database/pinecone\";\n",
        "import { Document } from \"@langchain/core/documents\";\n",
        "\n",
        "const embeddings = new OpenAIEmbeddings();\n",
        "\n",
        "const pinecone = new Pinecone();\n",
        "\n",
        "const pineconeIndex = pinecone.Index(process.env.PINECONE_INDEX);\n",
        "\n",
        "/**\n",
        " * Pinecone allows you to partition the records in an index into namespaces. \n",
        " * Queries and other operations are then limited to one namespace, \n",
        " * so different requests can search different subsets of your index.\n",
        " * Read more about namespaces here: https://docs.pinecone.io/guides/indexes/use-namespaces\n",
        " * \n",
        " * NOTE: If you have namespace enabled in your Pinecone index, you must provide the namespace when creating the PineconeStore.\n",
        " */\n",
        "const namespace = \"pinecone\";\n",
        "\n",
        "const vectorStore = await PineconeStore.fromExistingIndex(\n",
        "  new OpenAIEmbeddings(),\n",
        "  { pineconeIndex, namespace },\n",
        ");\n",
        "\n",
        "await vectorStore.addDocuments(\n",
        "  [new Document({ pageContent: \"i worked at kensho\" })],\n",
        "  { namespace: \"harrison\" },\n",
        ");\n",
        "\n",
        "await vectorStore.addDocuments(\n",
        "  [new Document({ pageContent: \"i worked at facebook\" })],\n",
        "  { namespace: \"ankush\" },\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "39c11920",
      "metadata": {},
      "source": [
        "The pinecone kwarg for `namespace` can be used to separate documents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "3c2a39fa",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[ Document { pageContent: \u001b[32m\"i worked at facebook\"\u001b[39m, metadata: {} } ]"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "// This will only get documents for Ankush\n",
        "const ankushRetriever = vectorStore.asRetriever({\n",
        "  filter: {\n",
        "    namespace: \"ankush\",\n",
        "  },\n",
        "});\n",
        "\n",
        "await ankushRetriever.invoke(\n",
        "  \"where did i work?\",\n",
        ");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "56393baa",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[ Document { pageContent: \u001b[32m\"i worked at kensho\"\u001b[39m, metadata: {} } ]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "// This will only get documents for Harrison\n",
        "const harrisonRetriever = vectorStore.asRetriever({\n",
        "  filter: {\n",
        "    namespace: \"harrison\",\n",
        "  },\n",
        "});\n",
        "\n",
        "await harrisonRetriever.invoke(\n",
        "  \"where did i work?\",\n",
        ");"
      ]
    },
    {
      "cell_type": "markdown",


{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# How to add chat history\n",
        "\n",
        "\n",
        ":::note\n",
        "\n",
        "This tutorial previously built a chatbot using [RunnableWithMessageHistory](https://api.js.langchain.com/classes/_langchain_core.runnables.RunnableWithMessageHistory.html). You can access this version of the tutorial in the [v0.2 docs](https://js.langchain.com/v0.2/docs/how_to/qa_chat_history_how_to/).\n",
        "\n",
        "The LangGraph implementation offers a number of advantages over `RunnableWithMessageHistory`, including the ability to persist arbitrary components of an application's state (instead of only messages).\n",
        "\n",
        ":::\n",
        "\n",
        "In many Q&A applications we want to allow the user to have a back-and-forth conversation, meaning the application needs some sort of \"memory\" of past questions and answers, and some logic for incorporating those into its current thinking.\n",
        "\n",
        "In this guide we focus on **adding logic for incorporating historical messages.**\n",
        "\n",
        "This is largely a condensed version of the [Conversational RAG tutorial](/docs/tutorials/qa_chat_history).\n",
        "\n",
        "We will cover two approaches:\n",
        "\n",
        "1. [Chains](/docs/how_to/qa_chat_history_how_to#chains), in which we always execute a retrieval step;\n",
        "2. [Agents](/docs/how_to/qa_chat_history_how_to#agents), in which we give an LLM discretion over whether and how to execute a retrieval step (or multiple steps).\n",
        "\n",
        "For the external knowledge source, we will use the same [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng from the [RAG tutorial](/docs/tutorials/rag)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setup\n",
        "### Dependencies\n",
        "\n",
        "We’ll use an OpenAI chat model and embeddings and a Memory vector store in this walkthrough, but everything shown here works with any [ChatModel](/docs/concepts/#chat-models) or [LLM](/docs/concepts#llms), [Embeddings](/docs/concepts#embedding-models), and [VectorStore](/docs/concepts#vectorstores) or [Retriever](/docs/concepts#retrievers).\n",
        "\n",
        "We’ll use the following packages:\n",
        "\n",
        "```bash\n",
        "npm install --save langchain @langchain/openai langchain cheerio uuid\n",
        "```\n",
        "\n",
        "We need to set environment variable `OPENAI_API_KEY`:\n",
        "\n",
        "```bash\n",
        "export OPENAI_API_KEY=YOUR_KEY\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### LangSmith\n",
        "\n",
        "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with [LangSmith](https://docs.smith.langchain.com).\n",
        "\n",
        "Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
        "\n",
        "\n",
        "```bash\n",
        "export LANGCHAIN_TRACING_V2=true\n",
        "export LANGCHAIN_API_KEY=YOUR_KEY\n",
        "\n",
        "# Reduce tracing latency if you are not in a serverless environment\n",
        "# export LANGCHAIN_CALLBACKS_BACKGROUND=true\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Chains {#chains}\n",
        "\n",
        "In a conversational RAG application, queries issued to the retriever should be informed by the context of the conversation. LangChain provides a [createHistoryAwareRetriever](https://api.js.langchain.com/functions/langchain.chains_history_aware_retriever.createHistoryAwareRetriever.html) constructor to simplify this. It constructs a chain that accepts keys `input` and `chat_history` as input, and has the same output schema as a retriever. `createHistoryAwareRetriever` requires as inputs:  \n",
        "\n",
        "1. LLM;\n",
        "2. Retriever;\n",
        "3. Prompt.\n",
        "\n",
        "First we obtain these objects:\n",
        "\n",
        "### LLM\n",
        "\n",
        "We can use any supported chat model:\n",
        "\n",
        "```{=mdx}\n",
        "import ChatModelTabs from \"@theme/ChatModelTabs\"\n",
        "\n",
        "<ChatModelTabs customVarName=\"llm\" />\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [],
      "source": [
        "// @lc-docs-hide-cell\n",
        "import { ChatOpenAI } from \"@langchain/openai\";\n",
        "\n",
        "const llm = new ChatOpenAI({ model: \"gpt-4o\" });"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Initial setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "import { CheerioWebBaseLoader } from \"@langchain/community/document_loaders/web/cheerio\";\n",
        "import { RecursiveCharacterTextSplitter } from \"langchain/text_splitter\";\n",
        "import { MemoryVectorStore } from \"langchain/vectorstores/memory\"\n",
        "import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
        "\n",
        "const loader = new CheerioWebBaseLoader(\n",
        "  \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\n",
        ");\n",
        "\n",
        "const docs = await loader.load();\n",
        "\n",
        "const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000, chunkOverlap: 200 });\n",
        "const splits = await textSplitter.splitDocuments(docs);\n",
        "const vectorStore = await MemoryVectorStore.fromDocuments(splits, new OpenAIEmbeddings());\n",
        "\n",
        "// Retrieve and generate using the relevant snippets of the blog.\n",
        "const retriever = vectorStore.asRetriever();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Prompt\n",
        "\n",
        "We'll use a prompt that includes a `MessagesPlaceholder` variable under the name \"chat_history\". This allows us to pass in a list of Messages to the prompt using the \"chat_history\" input key, and these messages will be inserted after the system message and before the human message containing the latest question."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "import { ChatPromptTemplate, MessagesPlaceholder } from \"@langchain/core/prompts\";\n",
        "\n",
        "const contextualizeQSystemPrompt = (\n",
        "  \"Given a chat history and the latest user question \" +\n",
        "  \"which might reference context in the chat history, \" +\n",
        "  \"formulate a standalone question which can be understood \" +\n",
        "  \"without the chat history. Do NOT answer the question, \" +\n",
        "  \"just reformulate it if needed and otherwise return it as is.\"\n",
        ")\n",
        "\n",
        "const contextualizeQPrompt = ChatPromptTemplate.fromMessages(\n",
        "  [\n",
        "    [\"system\", contextualizeQSystemPrompt],\n",
        "    new MessagesPlaceholder(\"chat_history\"),\n",
        "    [\"human\", \"{input}\"],\n",
        "  ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Assembling the chain\n",
        "\n",
        "We can then instantiate the history-aware retriever:"
      ]
    },


# How to load PDF files

> [Portable Document Format (PDF)](https://en.wikipedia.org/wiki/PDF), standardized as ISO 32000, is a file format developed by Adobe in 1992 to present documents, including text formatting and images, in a manner independent of application software, hardware, and operating systems.

This covers how to load `PDF` documents into the Document format that we use downstream.

By default, one document will be created for each page in the PDF file. You can change this behavior by setting the `splitPages` option to `false`.

## Setup

```bash npm2yarn
npm install pdf-parse
```

## Usage, one document per page

```typescript
import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
// Or, in web environments:
// import { WebPDFLoader } from "@langchain/community/document_loaders/web/pdf";
// const blob = new Blob(); // e.g. from a file input
// const loader = new WebPDFLoader(blob);

const loader = new PDFLoader("src/document_loaders/example_data/example.pdf");

const docs = await loader.load();
```

## Usage, one document per file

```typescript
import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";

const loader = new PDFLoader("src/document_loaders/example_data/example.pdf", {
  splitPages: false,
});

const docs = await loader.load();
```

## Usage, custom `pdfjs` build

By default we use the `pdfjs` build bundled with `pdf-parse`, which is compatible with most environments, including Node.js and modern browsers. If you want to use a more recent version of `pdfjs-dist` or if you want to use a custom build of `pdfjs-dist`, you can do so by providing a custom `pdfjs` function that returns a promise that resolves to the `PDFJS` object.

In the following example we use the "legacy" (see [pdfjs docs](https://github.com/mozilla/pdf.js/wiki/Frequently-Asked-Questions#which-browsersenvironments-are-supported)) build of `pdfjs-dist`, which includes several polyfills not included in the default build.

```bash npm2yarn
npm install pdfjs-dist
```

```typescript
import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";

const loader = new PDFLoader("src/document_loaders/example_data/example.pdf", {
  // you may need to add `.then(m => m.default)` to the end of the import
  pdfjs: () => import("pdfjs-dist/legacy/build/pdf.js"),
});
```

## Eliminating extra spaces

PDFs come in many varieties, which makes reading them a challenge. The loader parses individual text elements and joins them together with a space by default, but
if you are seeing excessive spaces, this may not be the desired behavior. In that case, you can override the separator with an empty string like this:

```typescript
import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";

const loader = new PDFLoader("src/document_loaders/example_data/example.pdf", {
  parsedItemSeparator: "",
});

const docs = await loader.load();
```


---
sidebar_position: 1
---

# How to stream responses from an LLM

All [`LLM`s](https://api.js.langchain.com/classes/langchain_core.language_models_llms.BaseLLM.html) implement the [Runnable interface](https://api.js.langchain.com/classes/langchain_core.runnables.Runnable.html), which comes with **default** implementations of standard runnable methods (i.e. `ainvoke`, `batch`, `abatch`, `stream`, `astream`, `astream_events`).

The **default** streaming implementations provide an `AsyncGenerator` that yields a single value: the final output from the underlying chat model provider.

The ability to stream the output token-by-token depends on whether the provider has implemented proper streaming support.

See which [integrations support token-by-token streaming here](/docs/integrations/llms/).

:::{.callout-note}

The **default** implementation does **not** provide support for token-by-token streaming, but it ensures that the model can be swapped in for any other model as it supports the same standard interface.

:::

## Using `.stream()`

import CodeBlock from "@theme/CodeBlock";

The easiest way to stream is to use the `.stream()` method. This returns an readable stream that you can also iterate over:

import StreamMethodExample from "@examples/models/llm/llm_streaming_stream_method.ts";

import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx";

<IntegrationInstallTooltip></IntegrationInstallTooltip>

```bash npm2yarn
npm install @langchain/openai @langchain/core
```

<CodeBlock language="typescript">{StreamMethodExample}</CodeBlock>

For models that do not support streaming, the entire response will be returned as a single chunk.

## Using a callback handler

You can also use a [`CallbackHandler`](https://api.js.langchain.com/classes/langchain_core.callbacks_base.BaseCallbackHandler.html) like so:

import StreamingExample from "@examples/models/llm/llm_streaming.ts";

<CodeBlock language="typescript">{StreamingExample}</CodeBlock>

We still have access to the end `LLMResult` if using `generate`. However, `tokenUsage` may not be currently supported for all model providers when streaming.