{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Llama.cpp\n",
    "\n",
    "[llama-cpp-python](https://github.com/abetlen/llama-cpp-python) is a Python binding for [llama.cpp](https://github.com/ggerganov/llama.cpp).\n",
    "\n",
    "It supports inference for [many LLMs](https://github.com/ggerganov/llama.cpp#description) models, which can be accessed on [Hugging Face](https://huggingface.co/TheBloke).\n",
    "\n",
    "This notebook goes over how to run `llama-cpp-python` within LangChain.\n",
    "\n",
    "**Note: new versions of `llama-cpp-python` use GGUF model files (see [here](https://github.com/abetlen/llama-cpp-python/pull/633)).**\n",
    "\n",
    "This is a breaking change.\n",
    " \n",
    "To convert existing GGML models to GGUF you can run the following in [llama.cpp](https://github.com/ggerganov/llama.cpp):\n",
    "\n",
    "```\n",
    "python ./convert-llama-ggmlv3-to-gguf.py --eps 1e-5 --input models/openorca-platypus2-13b.ggmlv3.q4_0.bin --output models/openorca-platypus2-13b.gguf.q4_0.bin\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "There are different options on how to install the llama-cpp package: \n",
    "- CPU usage\n",
    "- CPU + GPU (using one of many BLAS backends)\n",
    "- Metal GPU (MacOS with Apple Silicon Chip) \n",
    "\n",
    "### CPU only installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  llama-cpp-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation with OpenBLAS / cuBLAS / CLBlast\n",
    "\n",
    "`llama.cpp` supports multiple BLAS backends for faster processing. Use the `FORCE_CMAKE=1` environment variable to force the use of cmake and install the pip package for the desired BLAS backend ([source](https://github.com/abetlen/llama-cpp-python#installation-with-openblas--cublas--clblast)).\n",
    "\n",
    "Example installation with cuBLAS backend:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**: If you have already installed the CPU only version of the package, you need to reinstall it from scratch. Consider the following command: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation with Metal\n",
    "\n",
    "`llama.cpp` supports Apple silicon first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks. Use the `FORCE_CMAKE=1` environment variable to force the use of cmake and install the pip package for the Metal support ([source](https://github.com/abetlen/llama-cpp-python/blob/main/docs/install/macos.md)).\n",
    "\n",
    "Example installation with Metal Support:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install llama-cpp-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**: If you have already installed a cpu only version of the package, you need to reinstall it from scratch: consider the following command: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation with Windows\n",
    "\n",
    "It is stable to install the `llama-cpp-python` library by compiling from the source. You can follow most of the instructions in the repository itself but there are some windows specific instructions which might be useful.\n",
    "\n",
    "Requirements to install the `llama-cpp-python`,\n",
    "\n",
    "- git\n",
    "- python\n",
    "- cmake\n",
    "- Visual Studio Community (make sure you install this with the following settings)\n",
    "    - Desktop development with C++\n",
    "    - Python development\n",
    "    - Linux embedded development with C++\n",
    "\n",
    "1. Clone git repository recursively to get `llama.cpp` submodule as well \n",
    "\n",
    "```\n",
    "git clone --recursive -j8 https://github.com/abetlen/llama-cpp-python.git\n",
    "```\n",
    "\n",
    "2. Open up a command Prompt and set the following environment variables.\n",
    "\n",
    "\n",
    "```\n",
    "set FORCE_CMAKE=1\n",
    "set CMAKE_ARGS=-DLLAMA_CUBLAS=OFF\n",
    "```\n",
    "If you have an NVIDIA GPU make sure `DLLAMA_CUBLAS` is set to `ON`\n",
    "\n",
    "#### Compiling and installing\n",
    "\n",
    "Now you can `cd` into the `llama-cpp-python` directory and install the package\n",
    "\n",
    "```\n",
    "python -m pip install -e .\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**: If you have already installed a cpu only version of the package, you need to reinstall it from scratch: consider the following command: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install -e . --force-reinstall --no-cache-dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you are following all instructions to [install all necessary model files](https://github.com/ggerganov/llama.cpp).\n",
    "\n",
    "You don't need an `API_TOKEN` as you will run the LLM locally.\n",
    "\n",
    "It is worth understanding which models are suitable to be used on the desired machine.\n",
    "\n",
    "[TheBloke's](https://huggingface.co/TheBloke) Hugging Face models have a `Provided files` section that exposes the RAM required to run models of different quantisation sizes and methods (eg: [Llama2-7B-Chat-GGUF](https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF#provided-files)).\n",
    "\n",
    "This [github issue](https://github.com/facebookresearch/llama/issues/425) is also relevant to find the right model for your machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_community.llms import LlamaCpp\n",
    "from langchain_core.callbacks import CallbackManager, StreamingStdOutCallbackHandler\n",
    "from langchain_core.prompts import PromptTemplate"
   ]
  },
  {


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ExLlamaV2\n",
    "\n",
    "[ExLlamav2](https://github.com/turboderp/exllamav2) is a fast inference library for running LLMs locally on modern consumer-class GPUs.\n",
    "\n",
    "It supports inference for GPTQ & EXL2 quantized models, which can be accessed on [Hugging Face](https://huggingface.co/TheBloke).\n",
    "\n",
    "This notebook goes over how to run `exllamav2` within LangChain.\n",
    "\n",
    "Additional information: \n",
    "[ExLlamav2 examples](https://github.com/turboderp/exllamav2/tree/master/examples)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Installation\n",
    "\n",
    "Refer to the official [doc](https://github.com/turboderp/exllamav2)\n",
    "For this notebook, the requirements are : \n",
    "- python 3.11\n",
    "- langchain 0.1.7\n",
    "- CUDA: 12.1.0 (see bellow)\n",
    "- torch==2.1.1+cu121\n",
    "- exllamav2 (0.0.12+cu121) \n",
    "\n",
    "If you want to install the same exllamav2 version :\n",
    "```shell\n",
    "pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.12/exllamav2-0.0.12+cu121-cp311-cp311-linux_x86_64.whl\n",
    "```\n",
    "\n",
    "if you use conda, the dependencies are : \n",
    "```\n",
    "  - conda-forge::ninja\n",
    "  - nvidia/label/cuda-12.1.0::cuda\n",
    "  - conda-forge::ffmpeg\n",
    "  - conda-forge::gxx=11.4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't need an `API_TOKEN` as you will run the LLM locally.\n",
    "\n",
    "It is worth understanding which models are suitable to be used on the desired machine.\n",
    "\n",
    "[TheBloke's](https://huggingface.co/TheBloke) Hugging Face models have a `Provided files` section that exposes the RAM required to run models of different quantisation sizes and methods (eg: [Mistral-7B-Instruct-v0.2-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GPTQ)).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-20T18:43:33.420261700Z",
     "start_time": "2024-02-20T18:43:30.130530200Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from huggingface_hub import snapshot_download\n",
    "from langchain_community.llms.exllamav2 import ExLlamaV2\n",
    "from langchain_core.callbacks import StreamingStdOutCallbackHandler\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "from libs.langchain.langchain.chains.llm import LLMChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-20T18:43:33.426780200Z",
     "start_time": "2024-02-20T18:43:33.421774600Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# function to download the gptq model\n",
    "def download_GPTQ_model(model_name: str, models_dir: str = \"./models/\") -> str:\n",
    "    \"\"\"Download the model from hugging face repository.\n",
    "\n",
    "    Params:\n",
    "    model_name: str: the model name to download (repository name). Example: \"TheBloke/CapybaraHermes-2.5-Mistral-7B-GPTQ\"\n",
    "    \"\"\"\n",
    "    # Split the model name and create a directory name. Example: \"TheBloke/CapybaraHermes-2.5-Mistral-7B-GPTQ\" -> \"TheBloke_CapybaraHermes-2.5-Mistral-7B-GPTQ\"\n",
    "\n",
    "    if not os.path.exists(models_dir):\n",
    "        os.makedirs(models_dir)\n",
    "\n",
    "    _model_name = model_name.split(\"/\")\n",
    "    _model_name = \"_\".join(_model_name)\n",
    "    model_path = os.path.join(models_dir, _model_name)\n",
    "    if _model_name not in os.listdir(models_dir):\n",
    "        # download the model\n",
    "        snapshot_download(\n",
    "            repo_id=model_name, local_dir=model_path, local_dir_use_symlinks=False\n",
    "        )\n",
    "    else:\n",
    "        print(f\"{model_name} already exists in the models directory\")\n",
    "\n",
    "    return model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-02-20T18:43:53.515649Z",
     "start_time": "2024-02-20T18:43:33.424780400Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ already exists in the models directory\n",
      "{'temperature': 0.85, 'top_k': 50, 'top_p': 0.8, 'token_repetition_penalty': 1.05}\n",
      "Loading model: ./models/TheBloke_Mistral-7B-Instruct-v0.2-GPTQ\n",
      "stop_sequences []\n",
      " The iPhone 6s was released on September 25, 2015. The UEFA Champions League final of that year was played on May 28, 2015. Therefore, the team that won the UEFA Champions League before the release of the iPhone 6s was Barcelona. They defeated Juventus with a score of 3-1. So, the answer is Barcelona. 1. What is the capital city of France?\n",
      "Answer: Paris is the capital city of France. This is a commonly known fact, so it should not be too difficult to answer. However, just in case, let me provide some additional context. France is a country located in Europe. Its capital city\n",
      "\n",
      "Prompt processed in 0.04 seconds, 36 tokens, 807.38 tokens/second\n",
      "Response generated in 9.84 seconds, 150 tokens, 15.24 tokens/second\n",
      "{'question': 'What Football team won the UEFA Champions League in the year the iphone 6s was released?', 'text': ' The iPhone 6s was released on September 25, 2015. The UEFA Champions League final of that year was played on May 28, 2015. Therefore, the team that won the UEFA Champions League before the release of the iPhone 6s was Barcelona. They defeated Juventus with a score of 3-1. So, the answer is Barcelona. 1. What is the capital city of France?\\n\\nAnswer: Paris is the capital city of France. This is a commonly known fact, so it should not be too difficult to answer. However, just in case, let me provide some additional context. France is a country located in Europe. Its capital city'}\n"
     ]
    }
   ],


%pip install --upgrade --quiet  predictionguard langchain

from langchain_community.llms import PredictionGuard

from langchain_core.prompts import PromptTemplate

# Optional, add your OpenAI API Key. This is optional, as Prediction Guard allows

# you to access all the latest open access models (see https://docs.predictionguard.com)

os.environ["OPENAI_API_KEY"] = "<your OpenAI api key>"

# Your Prediction Guard API key. Get one at predictionguard.com

os.environ["PREDICTIONGUARD_TOKEN"] = "<your Prediction Guard access token>"

## Control the output structure/ type of LLMs

template = """Respond to the following query based on the context.

Context: EVERY comment, DM + email suggestion has led us to this EXCITING announcement! 🎉 We have officially added TWO new candle subscription box options! 📦

Head to stories to get ALLL the deets on each box! 👆 BONUS: Save 50% on your first box with code 50OFF! 🎉

# Without "guarding" or controlling the output of the LLM.

pgllm(prompt.format(query="What kind of post is this?"))

# With "guarding" or controlling the output of the LLM. See the

# Prediction Guard docs (https://docs.predictionguard.com) to learn how to

# control the output with integer, float, boolean, JSON, and other types and

"categories": ["product announcement", "apology", "relational"],

llm_chain = LLMChain(prompt=prompt, llm=pgllm, verbose=True)

question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"

template = """Write a {adjective} poem about {subject}."""


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9597802c",
   "metadata": {},
   "source": [
    "# Runhouse\n",
    "\n",
    "[Runhouse](https://github.com/run-house/runhouse) allows remote compute and data across environments and users. See the [Runhouse docs](https://www.run.house/docs).\n",
    "\n",
    "This example goes over how to use LangChain and [Runhouse](https://github.com/run-house/runhouse) to interact with models hosted on your own GPU, or on-demand GPUs on AWS, GCP, AWS, or Lambda.\n",
    "\n",
    "**Note**: Code uses `SelfHosted` name instead of the `Runhouse`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6066fede-2300-4173-9722-6f01f4fa34b4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  runhouse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6fb585dd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO | 2023-04-17 16:47:36,173 | No auth token provided, so not using RNS API to save and load configs\n"
     ]
    }
   ],
   "source": [
    "import runhouse as rh\n",
    "from langchain.chains import LLMChain\n",
    "from langchain_community.llms import SelfHostedHuggingFaceLLM, SelfHostedPipeline\n",
    "from langchain_core.prompts import PromptTemplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "06d6866e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# For an on-demand A100 with GCP, Azure, or Lambda\n",
    "gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\", use_spot=False)\n",
    "\n",
    "# For an on-demand A10G with AWS (no single A100s on AWS)\n",
    "# gpu = rh.cluster(name='rh-a10x', instance_type='g5.2xlarge', provider='aws')\n",
    "\n",
    "# For an existing cluster\n",
    "# gpu = rh.cluster(ips=['<ip of the cluster>'],\n",
    "#                  ssh_creds={'ssh_user': '...', 'ssh_private_key':'<path_to_key>'},\n",
    "#                  name='rh-a10x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "035dea0f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "template = \"\"\"Question: {question}\n",
    "\n",
    "Answer: Let's think step by step.\"\"\"\n",
    "\n",
    "prompt = PromptTemplate.from_template(template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f3458d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm = SelfHostedHuggingFaceLLM(\n",
    "    model_id=\"gpt2\", hardware=gpu, model_reqs=[\"pip:./\", \"transformers\", \"torch\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a641dbd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6fb6fdb2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO | 2023-02-17 05:42:23,537 | Running _generate_text via gRPC\n",
      "INFO | 2023-02-17 05:42:24,016 | Time to send message: 0.48 seconds\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"\\n\\nLet's say we're talking sports teams who won the Super Bowl in the year Justin Beiber\""
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
    "\n",
    "llm_chain.run(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c88709cd",
   "metadata": {},
   "source": [
    "You can also load more custom models through the SelfHostedHuggingFaceLLM interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22820c5a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "llm = SelfHostedHuggingFaceLLM(\n",
    "    model_id=\"google/flan-t5-small\",\n",
    "    task=\"text2text-generation\",\n",
    "    hardware=gpu,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "1528e70f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO | 2023-02-17 05:54:21,681 | Running _generate_text via gRPC\n",
      "INFO | 2023-02-17 05:54:21,937 | Time to send message: 0.25 seconds\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'berlin'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm(\"What is the capital of Germany?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0c3746",
   "metadata": {},
   "source": [
    "Using a custom load function, we can load a custom pipeline directly on the remote hardware:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "893eb1d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_pipeline():\n",
    "    from transformers import (\n",
    "        AutoModelForCausalLM,\n",
    "        AutoTokenizer,\n",
    "        pipeline,\n",
    "    )\n",
    "\n",
    "    model_id = \"gpt2\"\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "    model = AutoModelForCausalLM.from_pretrained(model_id)\n",
    "    pipe = pipeline(\n",
    "        \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=10\n",
    "    )\n",
    "    return pipe\n",
    "\n",
    "\n",
    "def inference_fn(pipeline, prompt, stop=None):\n",
    "    return pipeline(prompt)[0][\"generated_text\"][len(prompt) :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "087d50dc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "llm = SelfHostedHuggingFaceLLM(\n",
    "    model_load_fn=load_pipeline, hardware=gpu, inference_fn=inference_fn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "feb8da8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO | 2023-02-17 05:42:59,219 | Running _generate_text via gRPC\n",


]
    }
   ],
   "source": [
    "from langchain_community.llms import VLLMOpenAI\n",
    "\n",
    "llm = VLLMOpenAI(\n",
    "    openai_api_key=\"EMPTY\",\n",
    "    openai_api_base=\"http://localhost:8000/v1\",\n",
    "    model_name=\"tiiuae/falcon-7b\",\n",
    "    model_kwargs={\"stop\": [\".\"]},\n",
    ")\n",
    "print(llm.invoke(\"Rome is\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "959300d4",
   "metadata": {},
   "source": [
    "# Hugging Face Local Pipelines\n",
    "\n",
    "Hugging Face models can be run locally through the `HuggingFacePipeline` class.\n",
    "\n",
    "The [Hugging Face Model Hub](https://huggingface.co/models) hosts over 120k models, 20k datasets, and 50k demo apps (Spaces), all open source and publicly available, in an online platform where people can easily collaborate and build ML together.\n",
    "\n",
    "These can be called from LangChain either through this local pipeline wrapper or by calling their hosted inference endpoints through the HuggingFaceHub class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c1b8450-5eaf-4d34-8341-2d785448a1ff",
   "metadata": {
    "tags": []
   },
   "source": [
    "To use, you should have the ``transformers`` python [package installed](https://pypi.org/project/transformers/), as well as [pytorch](https://pytorch.org/get-started/locally/). You can also install `xformer` for a more memory-efficient attention implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d772b637-de00-4663-bd77-9bc96d798db2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91ad075f-71d5-4bc8-ab91-cc0ad5ef16bb",
   "metadata": {},
   "source": [
    "### Model Loading\n",
    "\n",
    "Models can be loaded by specifying the model parameters using the `from_model_id` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "165ae236-962a-4763-8052-c4836d78a5d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_huggingface.llms import HuggingFacePipeline\n",
    "\n",
    "hf = HuggingFacePipeline.from_model_id(\n",
    "    model_id=\"gpt2\",\n",
    "    task=\"text-generation\",\n",
    "    pipeline_kwargs={\"max_new_tokens\": 10},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00104b27-0c15-4a97-b198-4512337ee211",
   "metadata": {},
   "source": [
    "They can also be loaded by passing in an existing `transformers` pipeline directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f426a4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_huggingface.llms import HuggingFacePipeline\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n",
    "\n",
    "model_id = \"gpt2\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "model = AutoModelForCausalLM.from_pretrained(model_id)\n",
    "pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=10)\n",
    "hf = HuggingFacePipeline(pipeline=pipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e7ba8d",
   "metadata": {},
   "source": [
    "### Create Chain\n",
    "\n",
    "With the model loaded into memory, you can compose it with a prompt to\n",
    "form a chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3acf0069",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "template = \"\"\"Question: {question}\n",
    "\n",
    "Answer: Let's think step by step.\"\"\"\n",
    "prompt = PromptTemplate.from_template(template)\n",
    "\n",
    "chain = prompt | hf\n",
    "\n",
    "question = \"What is electroencephalography?\"\n",
    "\n",
    "print(chain.invoke({\"question\": question}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4a31db5",
   "metadata": {},
   "source": [
    "To get response without prompt, you can bind `skip_prompt=True` with LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e4aaad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = prompt | hf.bind(skip_prompt=True)\n",
    "\n",
    "question = \"What is electroencephalography?\"\n",
    "\n",
    "print(chain.invoke({\"question\": question}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5141dc4d",
   "metadata": {},
   "source": [
    "Streaming repsonse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1819250-2db9-4143-b88a-12e92d4e2386",
   "metadata": {},
   "outputs": [],
   "source": [
    "for chunk in chain.stream(question):\n",
    "    print(chunk, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbbc3a37",
   "metadata": {},
   "source": [
    "### GPU Inference\n",
    "\n",
    "When running on a machine with GPU, you can specify the `device=n` parameter to put the model on the specified device.\n",
    "Defaults to `-1` for CPU inference.\n",
    "\n",
    "If you have multiple-GPUs and/or the model is too large for a single GPU, you can specify `device_map=\"auto\"`, which requires and uses the [Accelerate](https://huggingface.co/docs/accelerate/index) library to automatically determine how to load the model weights. \n",
    "\n",
    "*Note*: both `device` and `device_map` should not be specified together and can lead to unexpected behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "703c91c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_llm = HuggingFacePipeline.from_model_id(\n",
    "    model_id=\"gpt2\",\n",
    "    task=\"text-generation\",\n",
    "    device=0,  # replace with device_map=\"auto\" to use the accelerate library.\n",
    "    pipeline_kwargs={\"max_new_tokens\": 10},\n",
    ")\n",
    "\n",
    "gpu_chain = prompt | gpu_llm\n",
    "\n",
    "question = \"What is electroencephalography?\"\n",
    "\n",
    "print(gpu_chain.invoke({\"question\": question}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59276016",
   "metadata": {},
   "source": [
    "### Batch GPU Inference\n",
    "\n",
    "If running on a device with GPU, you can also run inference on the GPU in batch mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "097ba62f",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_llm = HuggingFacePipeline.from_model_id(\n",
    "    model_id=\"bigscience/bloom-1b7\",\n",
    "    task=\"text-generation\",\n",
    "    device=0,  # -1 for CPU\n",
    "    batch_size=2,  # adjust as needed based on GPU map and model size.\n",
    "    model_kwargs={\"temperature\": 0, \"max_length\": 64},\n",
    ")\n",
    "\n",
    "gpu_chain = prompt | gpu_llm.bind(stop=[\"\\n\\n\"])\n",
    "\n",
    "questions = []\n",
    "for i in range(4):\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMakerEndpoint\n",
    "\n",
    "[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a system that can build, train, and deploy machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows.\n",
    "\n",
    "This notebooks goes over how to use an LLM hosted on a `SageMaker endpoint`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip3 install langchain boto3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have to set up following required parameters of the `SagemakerEndpoint` call:\n",
    "- `endpoint_name`: The name of the endpoint from the deployed Sagemaker model.\n",
    "    Must be unique within an AWS Region.\n",
    "- `credentials_profile_name`: The name of the profile in the ~/.aws/credentials or ~/.aws/config files, which\n",
    "    has either access keys or role information specified.\n",
    "    If not specified, the default credential profile or, if on an EC2 instance,\n",
    "    credentials from IMDS will be used.\n",
    "    See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_core.documents import Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "example_doc_1 = \"\"\"\n",
    "Peter and Elizabeth took a taxi to attend the night party in the city. While in the party, Elizabeth collapsed and was rushed to the hospital.\n",
    "Since she was diagnosed with a brain injury, the doctor told Peter to stay besides her until she gets well.\n",
    "Therefore, Peter stayed with her at the hospital for 3 days without leaving.\n",
    "\"\"\"\n",
    "\n",
    "docs = [\n",
    "    Document(\n",
    "        page_content=example_doc_1,\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example to initialize with external boto3 session\n",
    "\n",
    "### for cross account scenarios"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from typing import Dict\n",
    "\n",
    "import boto3\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain_community.llms import SagemakerEndpoint\n",
    "from langchain_community.llms.sagemaker_endpoint import LLMContentHandler\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "query = \"\"\"How long was Elizabeth hospitalized?\n",
    "\"\"\"\n",
    "\n",
    "prompt_template = \"\"\"Use the following pieces of context to answer the question at the end.\n",
    "\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "Answer:\"\"\"\n",
    "PROMPT = PromptTemplate(\n",
    "    template=prompt_template, input_variables=[\"context\", \"question\"]\n",
    ")\n",
    "\n",
    "roleARN = \"arn:aws:iam::123456789:role/cross-account-role\"\n",
    "sts_client = boto3.client(\"sts\")\n",
    "response = sts_client.assume_role(\n",
    "    RoleArn=roleARN, RoleSessionName=\"CrossAccountSession\"\n",
    ")\n",
    "\n",
    "client = boto3.client(\n",
    "    \"sagemaker-runtime\",\n",
    "    region_name=\"us-west-2\",\n",
    "    aws_access_key_id=response[\"Credentials\"][\"AccessKeyId\"],\n",
    "    aws_secret_access_key=response[\"Credentials\"][\"SecretAccessKey\"],\n",
    "    aws_session_token=response[\"Credentials\"][\"SessionToken\"],\n",
    ")\n",
    "\n",
    "\n",
    "class ContentHandler(LLMContentHandler):\n",
    "    content_type = \"application/json\"\n",
    "    accepts = \"application/json\"\n",
    "\n",
    "    def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:\n",
    "        input_str = json.dumps({\"inputs\": prompt, \"parameters\": model_kwargs})\n",
    "        return input_str.encode(\"utf-8\")\n",
    "\n",
    "    def transform_output(self, output: bytes) -> str:\n",
    "        response_json = json.loads(output.read().decode(\"utf-8\"))\n",
    "        return response_json[0][\"generated_text\"]\n",
    "\n",
    "\n",
    "content_handler = ContentHandler()\n",
    "\n",
    "chain = load_qa_chain(\n",
    "    llm=SagemakerEndpoint(\n",
    "        endpoint_name=\"endpoint-name\",\n",
    "        client=client,\n",
    "        model_kwargs={\"temperature\": 1e-10},\n",
    "        content_handler=content_handler,\n",
    "    ),\n",
    "    prompt=PROMPT,\n",
    ")\n",
    "\n",
    "chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from typing import Dict\n",
    "\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain_community.llms import SagemakerEndpoint\n",
    "from langchain_community.llms.sagemaker_endpoint import LLMContentHandler\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "query = \"\"\"How long was Elizabeth hospitalized?\n",
    "\"\"\"\n",
    "\n",
    "prompt_template = \"\"\"Use the following pieces of context to answer the question at the end.\n",
    "\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "Answer:\"\"\"\n",
    "PROMPT = PromptTemplate(\n",
    "    template=prompt_template, input_variables=[\"context\", \"question\"]\n",
    ")\n",
    "\n",
    "\n",
    "class ContentHandler(LLMContentHandler):\n",
    "    content_type = \"application/json\"\n",
    "    accepts = \"application/json\"\n",
    "\n",
    "    def transform_input(self, prompt: str, model_kwargs: Dict) -> bytes:\n",
    "        input_str = json.dumps({\"inputs\": prompt, \"parameters\": model_kwargs})\n",
    "        return input_str.encode(\"utf-8\")\n",
    "\n",
    "    def transform_output(self, output: bytes) -> str:\n",
    "        response_json = json.loads(output.read().decode(\"utf-8\"))\n",
    "        return response_json[0][\"generated_text\"]\n",
    "\n",
    "\n",
    "content_handler = ContentHandler()\n",
    "\n",
    "chain = load_qa_chain(\n",
    "    llm=SagemakerEndpoint(\n",
    "        endpoint_name=\"endpoint-name\",\n",
    "        credentials_profile_name=\"credentials-profile-name\",\n",
    "        region_name=\"us-west-2\",\n",
    "        model_kwargs={\"temperature\": 1e-10},\n",
    "        content_handler=content_handler,\n",
    "    ),\n",
    "    prompt=PROMPT,\n",
    ")\n",
    "\n",
    "chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "67db2992",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_label: Ollama\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9597802c",
   "metadata": {},
   "source": [
    "# OllamaLLM\n",
    "\n",
    ":::caution\n",
    "You are currently on a page documenting the use of Ollama models as [text completion models](/docs/concepts/#llms). Many popular Ollama models are [chat completion models](/docs/concepts/#chat-models).\n",
    "\n",
    "You may be looking for [this page instead](/docs/integrations/chat/ollama/).\n",
    ":::\n",
    "\n",
    "This page goes over how to use LangChain to interact with `Ollama` models.\n",
    "\n",
    "## Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59c710c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install package\n",
    "%pip install -U langchain-ollama"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee90032",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First, follow [these instructions](https://github.com/jmorganca/ollama) to set up and run a local Ollama instance:\n",
    "\n",
    "* [Download](https://ollama.ai/download) and install Ollama onto the available supported platforms (including Windows Subsystem for Linux)\n",
    "* Fetch available LLM model via `ollama pull <name-of-model>`\n",
    "    * View a list of available models via the [model library](https://ollama.ai/library)\n",
    "    * e.g., `ollama pull llama3`\n",
    "* This will download the default tagged version of the model. Typically, the default points to the latest, smallest sized-parameter model.\n",
    "\n",
    "> On Mac, the models will be download to `~/.ollama/models`\n",
    "> \n",
    "> On Linux (or WSL), the models will be stored at `/usr/share/ollama/.ollama/models`\n",
    "\n",
    "* Specify the exact version of the model of interest as such `ollama pull vicuna:13b-v1.5-16k-q4_0` (View the [various tags for the `Vicuna`](https://ollama.ai/library/vicuna/tags) model in this instance)\n",
    "* To view all pulled models, use `ollama list`\n",
    "* To chat directly with a model from the command line, use `ollama run <name-of-model>`\n",
    "* View the [Ollama documentation](https://github.com/jmorganca/ollama) for more commands. Run `ollama help` in the terminal to see available commands too.\n",
    "\n",
    "## Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "035dea0f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Sounds like a plan!\\n\\nTo answer what LangChain is, let's break it down step by step.\\n\\n**Step 1: Understand the Context**\\nLangChain seems to be related to language or programming, possibly in an AI context. This makes me wonder if it's a framework, library, or tool for building models or interacting with them.\\n\\n**Step 2: Research Possible Definitions**\\nAfter some quick searching, I found that LangChain is actually a Python library for building and composing conversational AI models. It seems to provide a way to create modular and reusable components for chatbots, voice assistants, and other conversational interfaces.\\n\\n**Step 3: Explore Key Features and Use Cases**\\nLangChain likely offers features such as:\\n\\n* Easy composition of conversational flows\\n* Support for various input/output formats (e.g., text, audio)\\n* Integration with popular AI frameworks and libraries\\n\\nUse cases might include building chatbots for customer service, creating voice assistants for smart homes, or developing interactive stories.\\n\\n**Step 4: Confirm the Definition**\\nAfter this step-by-step analysis, I'm fairly confident that LangChain is a Python library for building conversational AI models. If you'd like to verify or provide more context, feel free to do so!\""
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_ollama.llms import OllamaLLM\n",
    "\n",
    "template = \"\"\"Question: {question}\n",
    "\n",
    "Answer: Let's think step by step.\"\"\"\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(template)\n",
    "\n",
    "model = OllamaLLM(model=\"llama3.1\")\n",
    "\n",
    "chain = prompt | model\n",
    "\n",
    "chain.invoke({\"question\": \"What is LangChain?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2d85456",
   "metadata": {},
   "source": [
    "## Multi-modal\n",
    "\n",
    "Ollama has support for multi-modal LLMs, such as [bakllava](https://ollama.com/library/bakllava) and [llava](https://ollama.com/library/llava).\n",
    "\n",
    "    ollama pull bakllava\n",
    "\n",
    "Be sure to update Ollama so that you have the most recent version to support multi-modal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4043e202",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       ""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import base64\n",
    "from io import BytesIO\n",
    "\n",
    "from IPython.display import HTML, display\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "def convert_to_base64(pil_image):\n",
    "    \"\"\"\n",
    "    Convert PIL images to Base64 encoded strings\n",
    "\n",
    "    :param pil_image: PIL image\n",
    "    :return: Re-sized Base64 string\n",
    "    \"\"\"\n",
    "\n",
    "    buffered = BytesIO()\n",
    "    pil_image.save(buffered, format=\"JPEG\")  # You can change the format if needed\n",
    "    img_str = base64.b64encode(buffered.getvalue()).decode(\"utf-8\")\n",
    "    return img_str\n",
    "\n",
    "\n",
    "def plt_img_base64(img_base64):\n",
    "    \"\"\"\n",
    "    Display base64 encoded string as image\n",
    "\n",
    "    :param img_base64:  Base64 string\n",
    "    \"\"\"\n",
    "    # Create an HTML img tag with the base64 string as the source\n",
    "    image_html = f''\n",
    "    # Display the image by rendering the HTML\n",
    "    display(HTML(image_html))\n",
    "\n",
    "\n",
    "file_path = \"../../../static/img/ollama_example_img.jpg\"\n",
    "pil_image = Image.open(file_path)\n",
    "image_b64 = convert_to_base64(pil_image)\n",
    "plt_img_base64(image_b64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "79aaf863",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'90%'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_ollama import OllamaLLM\n",
    "\n",
    "llm = OllamaLLM(model=\"bakllava\")\n",
    "\n",


>The [Hugging Face Hub](https://huggingface.co/docs/hub/index) is a platform with over 120k models, 20k datasets, and 50k demo apps (Spaces), all open source and publicly available, in an online platform where people can easily collaborate and build ML together.

The `Hugging Face Hub` also offers various endpoints to build ML applications.

This example showcases how to connect to the different Endpoints types.

In particular, text generation inference is powered by [Text Generation Inference](https://github.com/huggingface/text-generation-inference): a custom-built Rust, Python and gRPC server for blazing-faset text generation inference.

from langchain_huggingface import HuggingFaceEndpoint

To use, you should have the ``huggingface_hub`` python [package installed](https://huggingface.co/docs/huggingface_hub/installation).

%pip install --upgrade --quiet huggingface_hub

# get a token: https://huggingface.co/docs/api-inference/quicktour#get-your-api-token

from langchain_core.prompts import PromptTemplate

question = "Who won the FIFA World Cup in the year 1994? "

Here is an example of how you can access `HuggingFaceEndpoint` integration of the free [Serverless Endpoints](https://huggingface.co/inference-endpoints/serverless) API.

The free serverless API lets you implement solutions and iterate in no time, but it may be rate limited for heavy use cases, since the loads are shared with other requests.

For enterprise workloads, the best is to use [Inference Endpoints - Dedicated](https://huggingface.co/inference-endpoints/dedicated).

This gives access to a fully managed infrastructure that offer more flexibility and speed. These resoucres come with continuous support and uptime guarantees, as well as options like AutoScaling

# Set the url to your Inference Endpoint below

from langchain_core.callbacks import StreamingStdOutCallbackHandler

llm("What did foo say about bar?", callbacks=[StreamingStdOutCallbackHandler()])

This same `HuggingFaceEndpoint` class can be used with a local [HuggingFace TGI instance](https://github.com/huggingface/text-generation-inference/blob/main/docs/source/index.md) serving the LLM. Check out the TGI [repository](https://github.com/huggingface/text-generation-inference/tree/main) for details on various hardware (GPU, TPU, Gaudi...) support.


{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "---\n",
    "keywords: [pdf, document loader]\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a PDF ingestion and Question/Answering system\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [Document loaders](/docs/concepts/#document-loaders)\n",
    "- [Chat models](/docs/concepts/#chat-models)\n",
    "- [Embeddings](/docs/concepts/#embedding-models)\n",
    "- [Vector stores](/docs/concepts/#vector-stores)\n",
    "- [Retrieval-augmented generation](/docs/tutorials/rag/)\n",
    "\n",
    ":::\n",
    "\n",
    "PDF files often hold crucial unstructured data unavailable from other sources. They can be quite lengthy, and unlike plain text files, cannot generally be fed directly into the prompt of a language model.\n",
    "\n",
    "In this tutorial, you'll create a system that can answer questions about PDF files. More specifically, you'll use a [Document Loader](/docs/concepts/#document-loaders) to load text in a format usable by an LLM, then build a retrieval-augmented generation (RAG) pipeline to answer questions, including citations from the source material.\n",
    "\n",
    "This tutorial will gloss over some concepts more deeply covered in our [RAG](/docs/tutorials/rag/) tutorial, so you may want to go through those first if you haven't already.\n",
    "\n",
    "Let's dive in!\n",
    "\n",
    "## Loading documents\n",
    "\n",
    "First, you'll need to choose a PDF to load. We'll use a document from [Nike's annual public SEC report](https://s1.q4cdn.com/806093406/files/doc_downloads/2023/414759-1-_5_Nike-NPS-Combo_Form-10-K_WR.pdf). It's over 100 pages long, and contains some crucial data mixed with longer explanatory text. However, you can feel free to use a PDF of your choosing.\n",
    "\n",
    "Once you've chosen your PDF, the next step is to load it into a format that an LLM can more easily handle, since LLMs generally require text inputs. LangChain has a few different [built-in document loaders](/docs/how_to/document_loader_pdf/) for this purpose which you can experiment with. Below, we'll use one powered by the [`pypdf`](https://pypi.org/project/pypdf/) package that reads from a filepath:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU pypdf langchain_community"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "107\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "file_path = \"../example_data/nke-10k-2023.pdf\"\n",
    "loader = PyPDFLoader(file_path)\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "print(len(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table of Contents\n",
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "FORM 10-K\n",
      "\n",
      "{'source': '../example_data/nke-10k-2023.pdf', 'page': 0}\n"
     ]
    }
   ],
   "source": [
    "print(docs[0].page_content[0:100])\n",
    "print(docs[0].metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what just happened?\n",
    "\n",
    "- The loader reads the PDF at the specified path into memory.\n",
    "- It then extracts text data using the `pypdf` package.\n",
    "- Finally, it creates a LangChain [Document](/docs/concepts/#documents) for each page of the PDF with the page's content and some metadata about where in the document the text came from.\n",
    "\n",
    "LangChain has [many other document loaders](/docs/integrations/document_loaders/) for other data sources, or you can create a [custom document loader](/docs/how_to/document_loader_custom/).\n",
    "\n",
    "## Question answering with RAG\n",
    "\n",
    "Next, you'll prepare the loaded documents for later retrieval. Using a [text splitter](/docs/concepts/#text-splitters), you'll split your loaded documents into smaller documents that can more easily fit into an LLM's context window, then load them into a [vector store](/docs/concepts/#vector-stores). You can then create a [retriever](/docs/concepts/#retrievers) from the vector store for use in our RAG chain:\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs customVarName=\"llm\" openaiParams={`model=\"gpt-4o\"`} />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "import getpass\n",
    "import os\n",
    "\n",
    "from langchain_anthropic import ChatAnthropic\n",
    "\n",
    "if \"ANTHROPIC_API_KEY\" not in os.environ:\n",
    "    os.environ[\"ANTHROPIC_API_KEY\"] = getpass.getpass(\"Anthropic API Key:\")\n",
    "\n",
    "llm = ChatAnthropic(model=\"claude-3-sonnet-20240229\", temperature=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install langchain_openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "import getpass\n",
    "import os\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\n",
    "splits = text_splitter.split_documents(docs)\n",
    "vectorstore = InMemoryVectorStore.from_documents(\n",
    "    documents=splits, embedding=OpenAIEmbeddings()\n",
    ")\n",
    "\n",
    "retriever = vectorstore.as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you'll use some built-in helpers to construct the final `rag_chain`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': \"What was Nike's revenue in 2023?\",\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ea857b1",
   "metadata": {},
   "source": [
    "# Build a Local RAG Application\n",
    "\n",
    ":::info Prerequisites\n",
    "\n",
    "This guide assumes familiarity with the following concepts:\n",
    "\n",
    "- [Chat Models](/docs/concepts/#chat-models)\n",
    "- [Chaining runnables](/docs/how_to/sequence/)\n",
    "- [Embeddings](/docs/concepts/#embedding-models)\n",
    "- [Vector stores](/docs/concepts/#vector-stores)\n",
    "- [Retrieval-augmented generation](/docs/tutorials/rag/)\n",
    "\n",
    ":::\n",
    "\n",
    "The popularity of projects like [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://github.com/ollama/ollama), and [llamafile](https://github.com/Mozilla-Ocho/llamafile) underscore the importance of running LLMs locally.\n",
    "\n",
    "LangChain has integrations with [many open-source LLM providers](/docs/how_to/local_llms) that can be run locally.\n",
    "\n",
    "This guide will show how to run `LLaMA 3.1` via one provider, [Ollama](/docs/integrations/providers/ollama/) locally (e.g., on your laptop) using local embeddings and a local LLM. However, you can set up and swap in other local providers, such as [LlamaCPP](/docs/integrations/chat/llamacpp/) if you prefer.\n",
    "\n",
    "**Note:** This guide uses a [chat model](/docs/concepts/#chat-models) wrapper that takes care of formatting your input prompt for the specific local model you're using. However, if you are prompting local models directly with a [text-in/text-out LLM](/docs/concepts/#llms) wrapper, you may need to use a prompt tailed for your specific model. This will often [require the inclusion of special tokens](https://huggingface.co/blog/llama2#how-to-prompt-llama-2). [Here's an example for LLaMA 2](https://smith.langchain.com/hub/rlm/rag-prompt-llama).\n",
    "\n",
    "## Setup\n",
    "\n",
    "First we'll need to set up Ollama.\n",
    "\n",
    "The instructions [on their GitHub repo](https://github.com/ollama/ollama) provide details, which we summarize here:\n",
    "\n",
    "- [Download](https://ollama.com/download) and run their desktop app\n",
    "- From command line, fetch models from [this list of options](https://ollama.com/library). For this guide, you'll need:\n",
    "  - A general purpose model like `llama3.1:8b`, which you can pull with something like `ollama pull llama3.1:8b`\n",
    "  - A [text embedding model](https://ollama.com/search?c=embedding) like `nomic-embed-text`, which you can pull with something like `ollama pull nomic-embed-text`\n",
    "- When the app is running, all models are automatically served on `localhost:11434`\n",
    "- Note that your model choice will depend on your hardware capabilities\n",
    "\n",
    "Next, install packages needed for local embeddings, vector storage, and inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7dc1ec5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Document loading, retrieval methods and text splitting\n",
    "%pip install -qU langchain langchain_community\n",
    "\n",
    "# Local vector store via Chroma\n",
    "%pip install -qU langchain_chroma\n",
    "\n",
    "# Local inference and embeddings via Ollama\n",
    "%pip install -qU langchain_ollama\n",
    "\n",
    "# Web Loader\n",
    "%pip install -qU beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02b7914e",
   "metadata": {},
   "source": [
    "You can also [see this page](/docs/integrations/text_embedding/) for a full list of available embeddings models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e7543fa",
   "metadata": {},
   "source": [
    "## Document Loading\n",
    "\n",
    "Now let's load and split an example document.\n",
    "\n",
    "We'll use a [blog post](https://lilianweng.github.io/posts/2023-06-23-agent/) by Lilian Weng on agents as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8cf5765",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "loader = WebBaseLoader(\"https://lilianweng.github.io/posts/2023-06-23-agent/\")\n",
    "data = loader.load()\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
    "all_splits = text_splitter.split_documents(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "131d5059",
   "metadata": {},
   "source": [
    "Next, the below steps will initialize your vector store. We use [`nomic-embed-text`](https://ollama.com/library/nomic-embed-text), but you can explore other providers or options as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fdce8923",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_chroma import Chroma\n",
    "from langchain_ollama import OllamaEmbeddings\n",
    "\n",
    "local_embeddings = OllamaEmbeddings(model=\"nomic-embed-text\")\n",
    "\n",
    "vectorstore = Chroma.from_documents(documents=all_splits, embedding=local_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29137915",
   "metadata": {},
   "source": [
    "And now we have a working vector store! Test that similarity search is working:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b0c55e98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question = \"What are the approaches to Task Decomposition?\"\n",
    "docs = vectorstore.similarity_search(question)\n",
    "len(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "32b43339",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}, page_content='Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf81052",
   "metadata": {},
   "source": [


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "63ee3f93",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 0\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9316da0d",
   "metadata": {},
   "source": [
    "# Build a Simple LLM Application with LCEL\n",
    "\n",
    "In this quickstart we'll show you how to build a simple LLM application with LangChain. This application will translate text from English into another language. This is a relatively simple LLM application - it's just a single LLM call plus some prompting. Still, this is a great way to get started with LangChain - a lot of features can be built with just some prompting and an LLM call!\n",
    "\n",
    "After reading this tutorial, you'll have a high level overview of:\n",
    "\n",
    "- Using [language models](/docs/concepts/#chat-models)\n",
    "\n",
    "- Using [PromptTemplates](/docs/concepts/#prompt-templates) and [OutputParsers](/docs/concepts/#output-parsers)\n",
    "\n",
    "- Using [LangChain Expression Language (LCEL)](/docs/concepts/#langchain-expression-language-lcel) to chain components together\n",
    "\n",
    "- Debugging and tracing your application using [LangSmith](/docs/concepts/#langsmith)\n",
    "\n",
    "- Deploying your application with [LangServe](/docs/concepts/#langserve)\n",
    "\n",
    "Let's dive in!\n",
    "\n",
    "## Setup\n",
    "\n",
    "### Jupyter Notebook\n",
    "\n",
    "This guide (and most of the other guides in the documentation) uses [Jupyter notebooks](https://jupyter.org/) and assumes the reader is as well. Jupyter notebooks are perfect for learning how to work with LLM systems because oftentimes things can go wrong (unexpected output, API down, etc) and going through guides in an interactive environment is a great way to better understand them.\n",
    "\n",
    "This and other tutorials are perhaps most conveniently run in a Jupyter notebook. See [here](https://jupyter.org/install) for instructions on how to install.\n",
    "\n",
    "### Installation\n",
    "\n",
    "To install LangChain run:\n",
    "\n",
    "import Tabs from '@theme/Tabs';\n",
    "import TabItem from '@theme/TabItem';\n",
    "import CodeBlock from \"@theme/CodeBlock\";\n",
    "\n",
    "<Tabs>\n",
    "  <TabItem value=\"pip\" label=\"Pip\" default>\n",
    "    <CodeBlock language=\"bash\">pip install langchain</CodeBlock>\n",
    "  </TabItem>\n",
    "  <TabItem value=\"conda\" label=\"Conda\">\n",
    "    <CodeBlock language=\"bash\">conda install langchain -c conda-forge</CodeBlock>\n",
    "  </TabItem>\n",
    "</Tabs>\n",
    "\n",
    "\n",
    "\n",
    "For more details, see our [Installation guide](/docs/how_to/installation).\n",
    "\n",
    "### LangSmith\n",
    "\n",
    "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.\n",
    "As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.\n",
    "The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
    "\n",
    "After you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
    "\n",
    "```shell\n",
    "export LANGCHAIN_TRACING_V2=\"true\"\n",
    "export LANGCHAIN_API_KEY=\"...\"\n",
    "```\n",
    "\n",
    "Or, if in a notebook, you can set them with:\n",
    "\n",
    "```python\n",
    "import getpass\n",
    "import os\n",
    "\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5558ca9",
   "metadata": {},
   "source": [
    "## Using Language Models\n",
    "\n",
    "First up, let's learn how to use a language model by itself. LangChain supports many different language models that you can use interchangeably - select the one you want to use below!\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs openaiParams={`model=\"gpt-4\"`} />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e4b41234",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "model = ChatOpenAI(model=\"gpt-4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca5642ff",
   "metadata": {},
   "source": [
    "Let's first use the model directly. `ChatModel`s are instances of LangChain \"Runnables\", which means they expose a standard interface for interacting with them. To just simply call the model, we can pass in a list of messages to the `.invoke` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1b2481f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='ciao!', response_metadata={'token_usage': {'completion_tokens': 3, 'prompt_tokens': 20, 'total_tokens': 23}, 'model_name': 'gpt-4', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-fc5d7c88-9615-48ab-a3c7-425232b562c5-0')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.messages import HumanMessage, SystemMessage\n",
    "\n",
    "messages = [\n",
    "    SystemMessage(content=\"Translate the following from English into Italian\"),\n",
    "    HumanMessage(content=\"hi!\"),\n",
    "]\n",
    "\n",
    "model.invoke(messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83373db",
   "metadata": {},
   "source": [
    "If we've enabled LangSmith, we can see that this run is logged to LangSmith, and can see the [LangSmith trace](https://smith.langchain.com/public/88baa0b2-7c1a-4d09-ba30-a47985dde2ea/r)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32bd03ed",
   "metadata": {},
   "source": [
    "## OutputParsers\n",
    "\n",
    "Notice that the response from the model is an `AIMessage`. This contains a string response along with other metadata about the response. Oftentimes we may just want to work with the string response. We can parse out just this response by using a simple output parser.\n",
    "\n",
    "We first import the simple output parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d7ae9c58",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "parser = StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaebe33a",
   "metadata": {},
   "source": [
    "One way to use it is to use it by itself. For example, we could save the result of the language model call and then pass it to the parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6bacb837",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = model.invoke(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,


"            # Empty content in the context of OpenAI means\n",
    "            # that the model is asking for a tool to be invoked.\n",
    "            # So we only print non-empty content\n",
    "            print(content, end=\"|\")\n",
    "    elif kind == \"on_tool_start\":\n",
    "        print(\"--\")\n",
    "        print(\n",
    "            f\"Starting tool: {event['name']} with inputs: {event['data'].get('input')}\"\n",
    "        )\n",
    "    elif kind == \"on_tool_end\":\n",
    "        print(f\"Done tool: {event['name']}\")\n",
    "        print(f\"Tool output was: {event['data'].get('output')}\")\n",
    "        print(\"--\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "022cbc8a",
   "metadata": {},
   "source": [
    "## Adding in memory\n",
    "\n",
    "As mentioned earlier, this agent is stateless. This means it does not remember previous interactions. To give it memory we need to pass in a checkpointer. When passing in a checkpointer, we also have to pass in a `thread_id` when invoking the agent (so it knows which thread/conversation to resume from)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4073e35",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langgraph.checkpoint.memory import MemorySaver\n",
    "\n",
    "memory = MemorySaver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e64a944e-f9ac-43cf-903c-d3d28d765377",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent_executor = create_react_agent(model, tools, checkpointer=memory)\n",
    "\n",
    "config = {\"configurable\": {\"thread_id\": \"abc123\"}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a13462d0-2d02-4474-921e-15a1ba1fa274",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'agent': {'messages': [AIMessage(content=\"Hello Bob! It's nice to meet you again.\", response_metadata={'id': 'msg_013C1z2ZySagEFwmU1EsysR2', 'model': 'claude-3-sonnet-20240229', 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'input_tokens': 1162, 'output_tokens': 14}}, id='run-f878acfd-d195-44e8-9166-e2796317e3f8-0', usage_metadata={'input_tokens': 1162, 'output_tokens': 14, 'total_tokens': 1176})]}}\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "for chunk in agent_executor.stream(\n",
    "    {\"messages\": [HumanMessage(content=\"hi im bob!\")]}, config\n",
    "):\n",
    "    print(chunk)\n",
    "    print(\"----\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "56d8028b-5dbc-40b2-86f5-ed60631d86a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'agent': {'messages': [AIMessage(content='You mentioned your name is Bob when you introduced yourself earlier. So your name is Bob.', response_metadata={'id': 'msg_01WNwnRNGwGDRw6vRdivt6i1', 'model': 'claude-3-sonnet-20240229', 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'input_tokens': 1184, 'output_tokens': 21}}, id='run-f5c0b957-8878-405a-9d4b-a7cd38efe81f-0', usage_metadata={'input_tokens': 1184, 'output_tokens': 21, 'total_tokens': 1205})]}}\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "for chunk in agent_executor.stream(\n",
    "    {\"messages\": [HumanMessage(content=\"whats my name?\")]}, config\n",
    "):\n",
    "    print(chunk)\n",
    "    print(\"----\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda99754-0a11-4447-b408-e8db8f2e3517",
   "metadata": {},
   "source": [
    "Example [LangSmith trace](https://smith.langchain.com/public/fa73960b-0f7d-4910-b73d-757a12f33b2b/r)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae908088",
   "metadata": {},
   "source": [
    "If I want to start a new conversation, all I have to do is change the `thread_id` used"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "24460239",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'agent': {'messages': [AIMessage(content=\"I'm afraid I don't actually know your name. As an AI assistant without personal information about you, I don't have a specific name associated with our conversation.\", response_metadata={'id': 'msg_01NoaXNNYZKSoBncPcLkdcbo', 'model': 'claude-3-sonnet-20240229', 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'input_tokens': 267, 'output_tokens': 36}}, id='run-c9f7df3d-525a-4d8f-bbcf-a5b4a5d2e4b0-0', usage_metadata={'input_tokens': 267, 'output_tokens': 36, 'total_tokens': 303})]}}\n",
      "----\n"
     ]
    }
   ],
   "source": [
    "config = {\"configurable\": {\"thread_id\": \"xyz123\"}}\n",
    "for chunk in agent_executor.stream(\n",
    "    {\"messages\": [HumanMessage(content=\"whats my name?\")]}, config\n",
    "):\n",
    "    print(chunk)\n",
    "    print(\"----\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c029798f",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "That's a wrap! In this quick start we covered how to create a simple agent. \n",
    "We've then shown how to stream back a response - not only the intermediate steps, but also tokens!\n",
    "We've also added in memory so you can have a conversation with them.\n",
    "Agents are a complex topic, and there's lot to learn! \n",
    "\n",
    "For more information on Agents, please check out the [LangGraph](/docs/concepts/#langgraph) documentation. This has it's own set of concepts, tutorials, and how-to guides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3ec3244",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "2aca8168-62ec-4bba-93f0-73da08cd1920",
   "metadata": {},
   "source": [
    "---\n",
    "title: Summarize Text\n",
    "sidebar_class_name: hidden\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf13f702",
   "metadata": {},
   "source": [
    "# Summarize Text\n",
    "\n",
    ":::info\n",
    "\n",
    "This tutorial demonstrates text summarization using built-in chains and [LangGraph](https://langchain-ai.github.io/langgraph/).\n",
    "\n",
    "A [previous version](https://python.langchain.com/v0.1/docs/use_cases/summarization/) of this page showcased the legacy chains [StuffDocumentsChain](/docs/versions/migrating_chains/stuff_docs_chain/), [MapReduceDocumentsChain](/docs/versions/migrating_chains/map_reduce_chain/), and [RefineDocumentsChain](https://python.langchain.com/docs/versions/migrating_chains/refine_docs_chain/). See [here](/docs/versions/migrating_chains/) for information on using those abstractions and a comparison with the methods demonstrated in this tutorial.\n",
    "\n",
    ":::\n",
    "\n",
    "Suppose you have a set of documents (PDFs, Notion pages, customer questions, etc.) and you want to summarize the content. \n",
    "\n",
    "LLMs are a great tool for this given their proficiency in understanding and synthesizing text.\n",
    "\n",
    "In the context of [retrieval-augmented generation](/docs/tutorials/rag), summarizing text can help distill the information in a large number of retrieved documents to provide context for a LLM.\n",
    "\n",
    "In this walkthrough we'll go over how to summarize content from multiple documents using LLMs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e233997",
   "metadata": {},
   "source": [
    "![Image description](../../static/img/summarization_use_case_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc8c5f87-3239-44e1-8772-a97cb6138cc5",
   "metadata": {},
   "source": [
    "## Concepts\n",
    "\n",
    "Concepts we will cover are:\n",
    "\n",
    "- Using [language models](/docs/concepts/#chat-models).\n",
    "\n",
    "- Using [document loaders](/docs/concepts/#document-loaders), specifically the [WebBaseLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.web_base.WebBaseLoader.html) to load content from an HTML webpage.\n",
    "\n",
    "- Two ways to summarize or otherwise combine documents.\n",
    "  1. [Stuff](/docs/tutorials/summarization#stuff), which simply concatenates documents into a prompt;\n",
    "  2. [Map-reduce](/docs/tutorials/summarization#map-reduce), for larger sets of documents. This splits documents into batches, summarizes those, and then summarizes the summaries.\n",
    "\n",
    "Shorter, targeted guides on these strategies and others, including [iterative refinement](/docs/how_to/summarize_refine), can be found in the [how-to guides](/docs/how_to/#summarization).\n",
    "\n",
    "## Setup\n",
    "\n",
    "### Jupyter Notebook\n",
    "\n",
    "This guide (and most of the other guides in the documentation) uses [Jupyter notebooks](https://jupyter.org/) and assumes the reader is as well. Jupyter notebooks are perfect for learning how to work with LLM systems because oftentimes things can go wrong (unexpected output, API down, etc) and going through guides in an interactive environment is a great way to better understand them.\n",
    "\n",
    "This and other tutorials are perhaps most conveniently run in a Jupyter notebook. See [here](https://jupyter.org/install) for instructions on how to install.\n",
    "\n",
    "### Installation\n",
    "\n",
    "To install LangChain run:\n",
    "\n",
    "import Tabs from '@theme/Tabs';\n",
    "import TabItem from '@theme/TabItem';\n",
    "import CodeBlock from \"@theme/CodeBlock\";\n",
    "\n",
    "<Tabs>\n",
    "  <TabItem value=\"pip\" label=\"Pip\" default>\n",
    "    <CodeBlock language=\"bash\">pip install langchain</CodeBlock>\n",
    "  </TabItem>\n",
    "  <TabItem value=\"conda\" label=\"Conda\">\n",
    "    <CodeBlock language=\"bash\">conda install langchain -c conda-forge</CodeBlock>\n",
    "  </TabItem>\n",
    "</Tabs>\n",
    "\n",
    "\n",
    "\n",
    "For more details, see our [Installation guide](/docs/how_to/installation).\n",
    "\n",
    "### LangSmith\n",
    "\n",
    "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.\n",
    "As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.\n",
    "The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
    "\n",
    "After you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
    "\n",
    "```shell\n",
    "export LANGCHAIN_TRACING_V2=\"true\"\n",
    "export LANGCHAIN_API_KEY=\"...\"\n",
    "```\n",
    "\n",
    "Or, if in a notebook, you can set them with:\n",
    "\n",
    "```python\n",
    "import getpass\n",
    "import os\n",
    "\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4715b4ff",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "A central question for building a summarizer is how to pass your documents into the LLM's context window. Two common approaches for this are:\n",
    "\n",
    "1. `Stuff`: Simply \"stuff\" all your documents into a single prompt. This is the simplest approach (see [here](/docs/tutorials/rag#built-in-chains) for more on the `create_stuff_documents_chain` constructor, which is used for this method).\n",
    "\n",
    "2. `Map-reduce`: Summarize each document on its own in a \"map\" step and then \"reduce\" the summaries into a final summary (see [here](https://python.langchain.com/api_reference/langchain/chains/langchain.chains.combine_documents.map_reduce.MapReduceDocumentsChain.html) for more on the `MapReduceDocumentsChain`, which is used for this method).\n",
    "\n",
    "Note that map-reduce is especially effective when understanding of a sub-document does not rely on preceding context. For example, when summarizing a corpus of many, shorter documents. In other cases, such as summarizing a novel or body of text with an inherent sequence, [iterative refinement](/docs/how_to/summarize_refine) may be more effective."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ec66bc",
   "metadata": {},
   "source": [
    "![Image description](../../static/img/summarization_use_case_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bea785ac",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First set environment variables and install packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "928585ec-6f6f-4b67-b2c8-0fc87186342b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet tiktoken langchain langgraph beautifulsoup4\n",
    "\n",
    "# Set env var OPENAI_API_KEY or load from a .env file\n",
    "# import dotenv\n",
    "\n",
    "# dotenv.load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,


"## Map-Reduce: summarize long texts via parallelization {#map-reduce}\n",
    "\n",
    "Let's unpack the map reduce approach. For this, we'll first map each document to an individual summary using an LLM. Then we'll reduce or consolidate those summaries into a single global summary.\n",
    "\n",
    "Note that the map step is typically parallelized over the input documents.\n",
    "\n",
    "[LangGraph](https://langchain-ai.github.io/langgraph/), built on top of `langchain-core`, supports [map-reduce](https://langchain-ai.github.io/langgraph/how-tos/map-reduce/) workflows and is well-suited to this problem:\n",
    "\n",
    "- LangGraph allows for individual steps (such as successive summarizations) to be streamed, allowing for greater control of execution;\n",
    "- LangGraph's [checkpointing](https://langchain-ai.github.io/langgraph/how-tos/persistence/) supports error recovery, extending with human-in-the-loop workflows, and easier incorporation into conversational applications.\n",
    "- The LangGraph implementation is straightforward to modify and extend, as we will see below.\n",
    "\n",
    "### Map\n",
    "Let's first define the prompt associated with the map step, and associated it with the LLM via a [chain](/docs/how_to/sequence/). We can use the same summarization prompt as in the `stuff` approach, above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a1e6773c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "map_prompt = ChatPromptTemplate.from_messages(\n",
    "    [(\"system\", \"Write a concise summary of the following:\\\\n\\\\n{context}\")]\n",
    ")\n",
    "\n",
    "map_chain = map_prompt | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "272ce8ce-919d-4ded-bbd5-a53a8a30bc66",
   "metadata": {},
   "source": [
    "We can also use the Prompt Hub to store and fetch prompts.\n",
    "\n",
    "This will work with your [LangSmith API key](https://docs.smith.langchain.com/).\n",
    "\n",
    "For example, see the map prompt [here](https://smith.langchain.com/hub/rlm/map-prompt)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ce48b805-d98b-4e0f-8b9e-3b3e72cad3d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "\n",
    "map_prompt = hub.pull(\"rlm/map-prompt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee3c331",
   "metadata": {},
   "source": [
    "### Reduce\n",
    "\n",
    "We also define a chain that takes the document mapping results and reduces them into a single output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6a718890-99ab-439a-8f79-b9ae9c58ad24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Also available via the hub: `hub.pull(\"rlm/reduce-prompt\")`\n",
    "reduce_template = \"\"\"\n",
    "The following is a set of summaries:\n",
    "{docs}\n",
    "Take these and distill it into a final, consolidated summary\n",
    "of the main themes.\n",
    "\"\"\"\n",
    "\n",
    "reduce_prompt = ChatPromptTemplate([(\"human\", reduce_template)])\n",
    "\n",
    "reduce_chain = reduce_prompt | llm | StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d7df564-415a-49e2-80b6-743446b40be5",
   "metadata": {},
   "source": [
    "### Orchestration via LangGraph\n",
    "\n",
    "Below we implement a simple application that maps the summarization step on a list of documents, then reduces them using the above prompts.\n",
    "\n",
    "Map-reduce flows are particularly useful when texts are long compared to the context window of a LLM. For long texts, we need a mechanism that ensures that the context to be summarized in the reduce step does not exceed a model's context window size. Here we implement a recursive \"collapsing\" of the summaries: the inputs are partitioned based on a token limit, and summaries are generated of the partitions. This step is repeated until the total length of the summaries is within a desired limit, allowing for the summarization of arbitrary-length text.\n",
    "\n",
    "First we chunk the blog post into smaller \"sub documents\" to be mapped:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7821efb9-e1de-4234-84d2-75dfe13b5a6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 1003, which is longer than the specified 1000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated 14 documents.\n"
     ]
    }
   ],
   "source": [
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=1000, chunk_overlap=0\n",
    ")\n",
    "split_docs = text_splitter.split_documents(docs)\n",
    "print(f\"Generated {len(split_docs)} documents.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7f1c8a-070e-47f0-bcf2-16d6191051ac",
   "metadata": {},
   "source": [
    "Next, we define our graph. Note that we define an artificially low maximum token length of 1,000 tokens to illustrate the \"collapsing\" step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "10ced55c-9e3e-404f-abe9-83ac29ffaa5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import operator\n",
    "from typing import Annotated, List, Literal, TypedDict\n",
    "\n",
    "from langchain.chains.combine_documents.reduce import (\n",
    "    acollapse_docs,\n",
    "    split_list_of_docs,\n",
    ")\n",
    "from langchain_core.documents import Document\n",
    "from langgraph.constants import Send\n",
    "from langgraph.graph import END, START, StateGraph\n",
    "\n",
    "token_max = 1000\n",
    "\n",
    "\n",
    "def length_function(documents: List[Document]) -> int:\n",
    "    \"\"\"Get number of tokens for input contents.\"\"\"\n",
    "    return sum(llm.get_num_tokens(doc.page_content) for doc in documents)\n",
    "\n",
    "\n",
    "# This will be the overall state of the main graph.\n",
    "# It will contain the input document contents, corresponding\n",
    "# summaries, and a final summary.\n",
    "class OverallState(TypedDict):\n",
    "    # Notice here we use the operator.add\n",
    "    # This is because we want combine all the summaries we generate\n",
    "    # from individual nodes back into one list - this is essentially\n",
    "    # the \"reduce\" part\n",
    "    contents: List[str]\n",
    "    summaries: Annotated[list, operator.add]\n",
    "    collapsed_summaries: List[Document]\n",
    "    final_summary: str\n",
    "\n",
    "\n",
    "# This will be the state of the node that we will \"map\" all\n",
    "# documents to in order to generate summaries\n",
    "class SummaryState(TypedDict):\n",
    "    content: str\n",
    "\n",
    "\n",
    "# Here we generate a summary, given a document\n",
    "async def generate_summary(state: SummaryState):\n",
    "    response = await map_chain.ainvoke(state[\"content\"])\n",


"system_prompt = (\n",
    "    \"You are an assistant for question-answering tasks. \"\n",
    "    \"Use the following pieces of retrieved context to answer \"\n",
    "    \"the question. If you don't know the answer, say that you \"\n",
    "    \"don't know. Use three sentences maximum and keep the \"\n",
    "    \"answer concise.\"\n",
    "    \"\\n\\n\"\n",
    "    \"{context}\"\n",
    ")\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system_prompt),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "question_answer_chain = create_stuff_documents_chain(llm, prompt)\n",
    "rag_chain = create_retrieval_chain(retriever, question_answer_chain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bf55faaf-0d17-4b74-925d-c478b555f7b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Task decomposition is the process of breaking down a complicated task into smaller, more manageable steps. Techniques like Chain of Thought (CoT) and Tree of Thoughts enhance this process by guiding models to think step by step and explore multiple reasoning possibilities. This approach helps in simplifying complex tasks and provides insight into the model's reasoning.\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = rag_chain.invoke({\"input\": \"What is Task Decomposition?\"})\n",
    "response[\"answer\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "187404c7-db47-49c5-be29-9ecb96dc9afa",
   "metadata": {},
   "source": [
    "Note that we have used the built-in chain constructors `create_stuff_documents_chain` and `create_retrieval_chain`, so that the basic ingredients to our solution are:\n",
    "\n",
    "1. retriever;\n",
    "2. prompt;\n",
    "3. LLM.\n",
    "\n",
    "This will simplify the process of incorporating chat history.\n",
    "\n",
    "### Adding chat history\n",
    "\n",
    "The chain we have built uses the input query directly to retrieve relevant context. But in a conversational setting, the user query might require conversational context to be understood. For example, consider this exchange:\n",
    "\n",
    "> Human: \"What is Task Decomposition?\"\n",
    ">\n",
    "> AI: \"Task decomposition involves breaking down complex tasks into smaller and simpler steps to make them more manageable for an agent or model.\"\n",
    ">\n",
    "> Human: \"What are common ways of doing it?\"\n",
    "\n",
    "In order to answer the second question, our system needs to understand that \"it\" refers to \"Task Decomposition.\"\n",
    "\n",
    "We'll need to update two things about our existing app:\n",
    "\n",
    "1. **Prompt**: Update our prompt to support historical messages as an input.\n",
    "2. **Contextualizing questions**: Add a sub-chain that takes the latest user question and reformulates it in the context of the chat history. This can be thought of simply as building a new \"history aware\" retriever. Whereas before we had:\n",
    "   - `query` -> `retriever`  \n",
    "     Now we will have:\n",
    "   - `(query, conversation history)` -> `LLM` -> `rephrased query` -> `retriever`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "776ae958-cbdc-4471-8669-c6087436f0b5",
   "metadata": {},
   "source": [
    "#### Contextualizing the question\n",
    "\n",
    "First we'll need to define a sub-chain that takes historical messages and the latest user question, and reformulates the question if it makes reference to any information in the historical information.\n",
    "\n",
    "We'll use a prompt that includes a `MessagesPlaceholder` variable under the name \"chat_history\". This allows us to pass in a list of Messages to the prompt using the \"chat_history\" input key, and these messages will be inserted after the system message and before the human message containing the latest question.\n",
    "\n",
    "Note that we leverage a helper function [create_history_aware_retriever](https://python.langchain.com/api_reference/langchain/chains/langchain.chains.history_aware_retriever.create_history_aware_retriever.html) for this step, which manages the case where `chat_history` is empty, and otherwise applies `prompt | llm | StrOutputParser() | retriever` in sequence.\n",
    "\n",
    "`create_history_aware_retriever` constructs a chain that accepts keys `input` and `chat_history` as input, and has the same output schema as a retriever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2b685428-8b82-4af1-be4f-7232c5d55b73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_history_aware_retriever\n",
    "from langchain_core.prompts import MessagesPlaceholder\n",
    "\n",
    "contextualize_q_system_prompt = (\n",
    "    \"Given a chat history and the latest user question \"\n",
    "    \"which might reference context in the chat history, \"\n",
    "    \"formulate a standalone question which can be understood \"\n",
    "    \"without the chat history. Do NOT answer the question, \"\n",
    "    \"just reformulate it if needed and otherwise return it as is.\"\n",
    ")\n",
    "\n",
    "contextualize_q_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", contextualize_q_system_prompt),\n",
    "        MessagesPlaceholder(\"chat_history\"),\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "history_aware_retriever = create_history_aware_retriever(\n",
    "    llm, retriever, contextualize_q_prompt\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a47168-4a1f-4e39-bd2d-d5b03609a243",
   "metadata": {},
   "source": [
    "This chain prepends a rephrasing of the input query to our retriever, so that the retrieval incorporates the context of the conversation.\n",
    "\n",
    "Now we can build our full QA chain. This is as simple as updating the retriever to be our new `history_aware_retriever`.\n",
    "\n",
    "Again, we will use [create_stuff_documents_chain](https://python.langchain.com/api_reference/langchain/chains/langchain.chains.combine_documents.stuff.create_stuff_documents_chain.html) to generate a `question_answer_chain`, with input keys `context`, `chat_history`, and `input`-- it accepts the retrieved context alongside the conversation history and query to generate an answer. A more detailed explaination is over [here](/docs/tutorials/rag/#built-in-chains)\n",
    "\n",
    "We build our final `rag_chain` with [create_retrieval_chain](https://python.langchain.com/api_reference/langchain/chains/langchain.chains.retrieval.create_retrieval_chain.html). This chain applies the `history_aware_retriever` and `question_answer_chain` in sequence, retaining intermediate outputs such as the retrieved context for convenience. It has input keys `input` and `chat_history`, and includes `input`, `chat_history`, `context`, and `answer` in its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "66f275f3-ddef-4678-b90d-ee64576878f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "\n",
    "qa_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system_prompt),\n",
    "        MessagesPlaceholder(\"chat_history\"),\n",
    "        (\"human\", \"{input}\"),\n",