{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6-0_o3DxsFGi"
   },
   "source": [
    "# Google Memorystore for Redis\n",
    "\n",
    "> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n",
    "\n",
    "This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to [save, load and delete langchain documents](/docs/how_to#document-loaders) with `MemorystoreDocumentLoader` and `MemorystoreDocumentSaver`.\n",
    "\n",
    "Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-memorystore-redis-python/).\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-memorystore-redis-python/blob/main/docs/document_loader.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before You Begin\n",
    "\n",
    "To run this notebook, you will need to do the following:\n",
    "\n",
    "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
    "* [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
    "* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 5.0.\n",
    "\n",
    "After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please specify an endpoint associated with the instance and a key prefix for demo purpose.\n",
    "ENDPOINT = \"redis://127.0.0.1:6379\"  # @param {type:\"string\"}\n",
    "KEY_PREFIX = \"doc:\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦜🔗 Library Installation\n",
    "\n",
    "The integration lives in its own `langchain-google-memorystore-redis` package, so we need to install it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -upgrade --quiet langchain-google-memorystore-redis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only**: Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☁ Set Your Google Cloud Project\n",
    "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n",
    "\n",
    "If you don't know your project ID, try the following:\n",
    "\n",
    "* Run `gcloud config list`.\n",
    "* Run `gcloud projects list`.\n",
    "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n",
    "\n",
    "PROJECT_ID = \"my-project-id\"  # @param {type:\"string\"}\n",
    "\n",
    "# Set the project id\n",
    "!gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🔐 Authentication\n",
    "\n",
    "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n",
    "\n",
    "- If you are using Colab to run this notebook, use the cell below and continue.\n",
    "- If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import auth\n",
    "\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2L7kMu__sFGl"
   },
   "source": [
    "## Basic Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save documents\n",
    "\n",
    "Save langchain documents with `MemorystoreDocumentSaver.add_documents(<documents>)`. To initialize `MemorystoreDocumentSaver` class you need to provide 2 things:\n",
    "\n",
    "1. `client` - A `redis.Redis` client object.\n",
    "1. `key_prefix` - A prefix for the keys to store Documents in Redis.\n",
    "\n",
    "The Documents will be stored into randomly generated keys with the specified prefix of `key_prefix`. Alternatively, you can designate the suffixes of the keys by specifying `ids` in the `add_documents` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import redis\n",
    "from langchain_core.documents import Document\n",
    "from langchain_google_memorystore_redis import MemorystoreDocumentSaver\n",
    "\n",
    "test_docs = [\n",
    "    Document(\n",
    "        page_content=\"Apple Granny Smith 150 0.99 1\",\n",
    "        metadata={\"fruit_id\": 1},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Banana Cavendish 200 0.59 0\",\n",
    "        metadata={\"fruit_id\": 2},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Orange Navel 80 1.29 1\",\n",
    "        metadata={\"fruit_id\": 3},\n",
    "    ),\n",
    "]\n",
    "doc_ids = [f\"{i}\" for i in range(len(test_docs))]\n",
    "\n",
    "redis_client = redis.from_url(ENDPOINT)\n",
    "saver = MemorystoreDocumentSaver(\n",
    "    client=redis_client,\n",
    "    key_prefix=KEY_PREFIX,\n",
    "    content_field=\"page_content\",\n",
    ")\n",
    "saver.add_documents(test_docs, ids=doc_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A2fT1iEhsFGl"
   },
   "source": [
    "### Load documents\n",
    "\n",
    "Initialize a loader that loads all documents stored in the Memorystore for Redis instance with a specific prefix.\n",
    "\n",
    "Load langchain documents with `MemorystoreDocumentLoader.load()` or `MemorystoreDocumentLoader.lazy_load()`. `lazy_load` returns a generator that only queries database during the iteration. To initialize `MemorystoreDocumentLoader` class you need to provide:\n",
    "\n",
    "1. `client` - A `redis.Redis` client object.\n",


>[Microsoft Word](https://www.microsoft.com/en-us/microsoft-365/word) is a word processor developed by Microsoft.

This covers how to load `Word` documents into a document format that we can use downstream.

Load .docx using `Docx2txt` into a document.

%pip install --upgrade --quiet  docx2txt

[Document(page_content='Lorem ipsum dolor sit amet.', metadata={'source': './example_data/fake.docx'})]

from langchain_community.document_loaders import Docx2txtLoader

Please see [this guide](/docs/integrations/providers/unstructured/) for more instructions on setting up Unstructured locally, including setting up required system dependencies.

[Document(page_content='Lorem ipsum dolor sit amet.', metadata={'source': 'example_data/fake.docx'})]

from langchain_community.document_loaders import UnstructuredWordDocumentLoader

Under the hood, Unstructured creates different "elements" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying `mode="elements"`.

Document(page_content='Lorem ipsum dolor sit amet.', metadata={'source': './example_data/fake.docx', 'category_depth': 0, 'file_directory': './example_data', 'filename': 'fake.docx', 'last_modified': '2023-12-19T13:42:18', 'languages': ['por', 'cat'], 'filetype': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'category': 'Title'})

loader = UnstructuredWordDocumentLoader("./example_data/fake.docx", mode="elements")

>[Azure AI Document Intelligence](https://aka.ms/doc-intelligence) (formerly known as `Azure Form Recognizer`) is machine-learning

>based service that extracts texts (including handwriting), tables, document structures (e.g., titles, section headings, etc.) and key-value-pairs from

>digital or scanned PDFs, images, Office and HTML files.

>Document Intelligence supports `PDF`, `JPEG/JPG`, `PNG`, `BMP`, `TIFF`, `HEIF`, `DOCX`, `XLSX`, `PPTX` and `HTML`.

This current implementation of a loader using `Document Intelligence` can incorporate content page-wise and turn it into LangChain documents. The default output format is markdown, which can be easily chained with `MarkdownHeaderTextSplitter` for semantic document chunking. You can also use `mode="single"` or `mode="page"` to return pure texts in a single page or document split by page.

An Azure AI Document Intelligence resource in one of the 3 preview regions: **East US**, **West US2**, **West Europe** - follow [this document](https://learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource?view=doc-intel-4.0.0) to create one if you don't have. You will be passing `<endpoint>` and `<key>` as parameters to the loader.

%pip install --upgrade --quiet  langchain langchain-community azure-ai-documentintelligence

from langchain_community.document_loaders import AzureAIDocumentIntelligenceLoader

api_endpoint=endpoint, api_key=key, file_path=file_path, api_model="prebuilt-layout"


This covers how to load images into a document format that we can use downstream with other LangChain modules.

It uses [Unstructured](https://unstructured.io/) to handle a wide variety of image formats, such as `.jpg` and `.png`. Please see [this guide](/docs/integrations/providers/unstructured/) for more instructions on setting up Unstructured locally, including setting up required system dependencies.

%pip install --upgrade --quiet "unstructured[all-docs]"

Document(page_content='2021\n\n2103.15348v2 [cs.CV] 21 Jun\n\narXiv\n\nLayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis\n\nZejiang Shen! (&4), Ruochen Zhang?, Melissa Dell?, Benjamin Charles Germain Lee*, Jacob Carlson?, and Weining Li?\n\n1\n\nAllen Institute for AI shannons@allenai.org ? Brown University ruochen_zhang@brown. edu 3 Harvard University {melissadell, jacob_carlson}@fas.harvard.edu 4 University of Washington begl@cs.washington.edu 5 University of Waterloo w4221i@uwaterloo.ca\n\nAbstract. Recent advances in document image analysis (DIA) have been primarily driven by the application of neural networks. Ideally, research outcomes could be easily deployed in production and extended for further investigation. However, various factors like loosely organized codebases and sophisticated model configurations complicate the easy reuse of im- portant innovations by a wide audience. Though there have been on-going efforts to improve reusability and simplify deep learning (DL) model development in disciplines like natural language processing and computer vision, none of them are optimized for challenges in the domain of DIA. This represents a major gap in the existing toolkit, as DIA is central to academic research across a wide range of disciplines in the social sciences and humanities. This paper introduces LayoutParser, an open-source library for streamlining the usage of DL in DIA research and applica- tions. The core LayoutParser library comes with a set of simple and intuitive interfaces for applying and customizing DL models for layout de- tection, character recognition, and many other document processing tasks. To promote extensibility, LayoutParser also incorporates a community platform for sharing both pre-trained models and full document digiti- zation pipelines. We demonstrate that LayoutParser is helpful for both lightweight and large-scale digitization pipelines in real-word use cases. The library is publicly available at https: //layout-parser.github. io.\n\nKeywords: Document Image Analysis - Deep Learning - Layout Analysis - Character Recognition - Open Source library - Toolkit.\n\n1 Introduction\n\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of document image analysis (DIA) tasks including document image classification [11,', metadata={'source': './example_data/layout-parser-paper-screenshot.png'})

from langchain_community.document_loaders.image import UnstructuredImageLoader

Under the hood, Unstructured creates different "elements" for different chunks of text. By default we combine those together, but you can keep that separation by specifying `mode="elements"`.

Document(page_content='2021', metadata={'source': './example_data/layout-parser-paper-screenshot.png', 'coordinates': {'points': ((47.0, 492.0), (47.0, 591.0), (83.0, 591.0), (83.0, 492.0)), 'system': 'PixelSpace', 'layout_width': 1624, 'layout_height': 1920}, 'last_modified': '2024-07-01T10:38:29', 'filetype': 'PNG', 'languages': ['eng'], 'page_number': 1, 'file_directory': './example_data', 'filename': 'layout-parser-paper-screenshot.png', 'category': 'UncategorizedText'})


"to get more details about configuration parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de97d0ed-d6b1-44e0-b392-1f3d89c762f9",
   "metadata": {},
   "source": [
    "### Basic example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "50ffeeee-db12-4801-b208-7e32ea3d72ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nMadam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \\n\\n\\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\n\\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\n\\n\\nWith a duty to one another to the American people to '"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.document_loaders import DedocFileLoader\n",
    "\n",
    "loader = DedocFileLoader(\"./example_data/state_of_the_union.txt\")\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "docs[0].page_content[:400]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "457e5d4c-a4ee-4f31-ae74-3f75a1bbd0af",
   "metadata": {},
   "source": [
    "### Modes of split\n",
    "\n",
    "`DedocFileLoader` supports different types of document splitting into parts (each part is returned separately).\n",
    "For this purpose, `split` parameter is used with the following options:\n",
    "* `document` (default value): document text is returned as a single langchain `Document` object (don't split);\n",
    "* `page`: split document text into pages (works for `PDF`, `DJVU`, `PPTX`, `PPT`, `ODP`);\n",
    "* `node`: split document text into `Dedoc` tree nodes (title nodes, list item nodes, raw text nodes);\n",
    "* `line`: split document text into textual lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "eec54d31-ae7a-4a3c-aa10-4ae276b1e4c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = DedocFileLoader(\n",
    "    \"./example_data/layout-parser-paper.pdf\",\n",
    "    split=\"page\",\n",
    "    pages=\":2\",\n",
    ")\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "len(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61e11769-4780-4f77-b10e-27db6936f226",
   "metadata": {},
   "source": [
    "### Handling tables\n",
    "\n",
    "`DedocFileLoader` supports tables handling when `with_tables` parameter is \n",
    "set to `True` during loader initialization (`with_tables=True` by default). \n",
    "\n",
    "Tables are not split - each table corresponds to one langchain `Document` object.\n",
    "For tables, `Document` object has additional `metadata` fields `type=\"table\"` \n",
    "and `text_as_html` with table `HTML` representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bbeb2f8a-ac5e-4b59-8026-7ea3fc14c928",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('table',\n",
       " '
\\n
\\n
\\nTeam	\\n "Payroll (millions)"	\\n<td colspan=\"1\" r')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = DedocFileLoader(\"./example_data/mlb_teams_2012.csv\")\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "docs[1].metadata[\"type\"], docs[1].metadata[\"text_as_html\"][:200]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4a2b872-2aba-4e4c-8b2f-83a5a81ee1da",
   "metadata": {},
   "source": [
    "### Handling attached files\n",
    "\n",
    "`DedocFileLoader` supports attached files handling when `with_attachments` is set \n",
    "to `True` during loader initialization (`with_attachments=False` by default). \n",
    "\n",
    "Attachments are split according to the `split` parameter.\n",
    "For attachments, langchain `Document` object has an additional metadata \n",
    "field `type=\"attachment\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bb9d6c1c-e24c-4979-88a0-38d54abd6332",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('attachment',\n",
       " '\\nContent-Type\\nmultipart/mixed; boundary=\"0000000000005d654405f082adb7\"\\nDate\\nFri, 23 Dec 2022 12:08:48 -0600\\nFrom\\nMallori Harrell <mallori@unstructured.io>\\nMIME-Version\\n1.0\\nMessage-ID\\n<CAPgNNXSzLVJ-d1OCX_TjFgJU7ugtQrjFybPtAMmmYZzphxNFYg@mail.gmail.com>\\nSubject\\nFake email with attachment\\nTo\\nMallori Harrell <mallori@unstructured.io>')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = DedocFileLoader(\n",
    "    \"./example_data/fake-email-attachment.eml\",\n",
    "    with_attachments=True,\n",
    ")\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "docs[1].metadata[\"type\"], docs[1].page_content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d435c3f6-703a-4064-8307-ace140de967a",
   "metadata": {},
   "source": [
    "## Loading PDF file\n",
    "\n",
    "If you want to handle only `PDF` documents, you can use `DedocPDFLoader` with only `PDF` support.\n",
    "The loader supports the same parameters for document split, tables and attachments extraction.\n",
    "\n",
    "`Dedoc` can extract `PDF` with or without a textual layer, \n",
    "as well as automatically detect its presence and correctness.\n",
    "Several `PDF` handlers are available, you can use `pdf_with_text_layer` \n",
    "parameter to choose one of them.\n",
    "Please see [parameters description](https://dedoc.readthedocs.io/en/latest/parameters/pdf_handling.html) \n",
    "to get more details.\n",
    "\n",
    "For `PDF` without a textual layer, `Tesseract OCR` and its language packages should be installed.\n",
    "In this case, [the instruction](https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html) can be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# JSONLoader\n",
    "\n",
    "This notebook provides a quick overview for getting started with JSON [document loader](https://python.langchain.com/docs/concepts/#document-loaders). For detailed documentation of all JSONLoader features and configurations head to the [API reference](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.json_loader.JSONLoader.html).\n",
    "\n",
    "- TODO: Add any other relevant links, like information about underlying API, etc.\n",
    "\n",
    "## Overview\n",
    "### Integration details\n",
    "\n",
    "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/document_loaders/file_loaders/json/)|\n",
    "| :--- | :--- | :---: | :---: |  :---: |\n",
    "| [JSONLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.json_loader.JSONLoader.html) | [langchain_community](https://python.langchain.com/api_reference/community/index.html) | ✅ | ❌ | ✅ | \n",
    "### Loader features\n",
    "| Source | Document Lazy Loading | Native Async Support\n",
    "| :---: | :---: | :---: | \n",
    "| JSONLoader | ✅ | ❌ | \n",
    "\n",
    "## Setup\n",
    "\n",
    "To access JSON document loader you'll need to install the `langchain-community` integration package as well as the ``jq`` python package.\n",
    "\n",
    "### Credentials\n",
    "\n",
    "No credentials are required to use the `JSONLoader` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to get automated best in-class tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",
    "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation\n",
    "\n",
    "Install **langchain_community** and **jq**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain_community jq "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "Now we can instantiate our model object and load documents:\n",
    "\n",
    "- TODO: Update model instantiation with relevant params."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import JSONLoader\n",
    "\n",
    "loader = JSONLoader(\n",
    "    file_path=\"./example_data/facebook_chat.json\",\n",
    "    jq_schema=\".messages[].content\",\n",
    "    text_content=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': '/Users/isaachershenson/Documents/langchain/docs/docs/integrations/document_loaders/example_data/facebook_chat.json', 'seq_num': 1}, page_content='Bye!')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = loader.load()\n",
    "docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': '/Users/isaachershenson/Documents/langchain/docs/docs/integrations/document_loaders/example_data/facebook_chat.json', 'seq_num': 1}\n"
     ]
    }
   ],
   "source": [
    "print(docs[0].metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lazy Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pages = []\n",
    "for doc in loader.lazy_load():\n",
    "    pages.append(doc)\n",
    "    if len(pages) >= 10:\n",
    "        # do some paged operation, e.g.\n",
    "        # index.upsert(pages)\n",
    "\n",
    "        pages = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read from JSON Lines file\n",
    "\n",
    "If you want to load documents from a JSON Lines file, you pass `json_lines=True`\n",
    "and specify `jq_schema` to extract `page_content` from a single JSON object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Bye!' metadata={'source': '/Users/isaachershenson/Documents/langchain/docs/docs/integrations/document_loaders/example_data/facebook_chat_messages.jsonl', 'seq_num': 1}\n"
     ]
    }
   ],
   "source": [
    "loader = JSONLoader(\n",
    "    file_path=\"./example_data/facebook_chat_messages.jsonl\",\n",
    "    jq_schema=\".content\",\n",
    "    text_content=False,\n",
    "    json_lines=True,\n",
    ")\n",
    "\n",
    "docs = loader.load()\n",
    "print(docs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read specific content keys\n",
    "\n",
    "Another option is to set `jq_schema='.'` and provide a `content_key` in order to only load specific content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='User 2' metadata={'source': '/Users/isaachershenson/Documents/langchain/docs/docs/integrations/document_loaders/example_data/facebook_chat_messages.jsonl', 'seq_num': 1}\n"
     ]
    }
   ],
   "source": [
    "loader = JSONLoader(\n",
    "    file_path=\"./example_data/facebook_chat_messages.jsonl\",\n",
    "    jq_schema=\".\",\n",
    "    content_key=\"sender_name\",\n",
    "    json_lines=True,\n",
    ")\n",
    "\n",
    "docs = loader.load()\n",
    "print(docs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON file with jq schema `content_key`\n",
    "\n",
    "To load documents from a JSON file using the `content_key` within the jq schema, set `is_content_key_jq_parsable=True`. Ensure that `content_key` is compatible and can be parsed using the jq schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Bye!' metadata={'source': '/Users/isaachershenson/Documents/langchain/docs/docs/integrations/document_loaders/example_data/facebook_chat.json', 'seq_num': 1}\n"
     ]
    }
   ],
   "source": [
    "loader = JSONLoader(\n",
    "    file_path=\"./example_data/facebook_chat.json\",\n",
    "    jq_schema=\".messages[]\",\n",
    "    content_key=\".content\",\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google Spanner\n",
    "\n",
    "> [Spanner](https://cloud.google.com/spanner) is a highly scalable database that combines unlimited scalability with relational semantics, such as secondary indexes, strong consistency, schemas, and SQL providing 99.999% availability in one easy solution.\n",
    "\n",
    "This notebook goes over how to use [Spanner](https://cloud.google.com/spanner) to [save, load and delete langchain documents](/docs/how_to#document-loaders) with `SpannerLoader` and `SpannerDocumentSaver`.\n",
    "\n",
    "Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-spanner-python/).\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-spanner-python/blob/main/docs/document_loader.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before You Begin\n",
    "\n",
    "To run this notebook, you will need to do the following:\n",
    "\n",
    "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
    "* [Enable the Cloud Spanner API](https://console.cloud.google.com/flows/enableapi?apiid=spanner.googleapis.com)\n",
    "* [Create a Spanner instance](https://cloud.google.com/spanner/docs/create-manage-instances)\n",
    "* [Create a Spanner database](https://cloud.google.com/spanner/docs/create-manage-databases)\n",
    "* [Create a Spanner table](https://cloud.google.com/spanner/docs/create-query-database-console#create-schema)\n",
    "\n",
    "After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please specify an instance id, a database, and a table for demo purpose.\n",
    "INSTANCE_ID = \"test_instance\"  # @param {type:\"string\"}\n",
    "DATABASE_ID = \"test_database\"  # @param {type:\"string\"}\n",
    "TABLE_NAME = \"test_table\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦜🔗 Library Installation\n",
    "\n",
    "The integration lives in its own `langchain-google-spanner` package, so we need to install it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install -upgrade --quiet langchain-google-spanner langchain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only**: Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☁ Set Your Google Cloud Project\n",
    "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n",
    "\n",
    "If you don't know your project ID, try the following:\n",
    "\n",
    "* Run `gcloud config list`.\n",
    "* Run `gcloud projects list`.\n",
    "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n",
    "\n",
    "PROJECT_ID = \"my-project-id\"  # @param {type:\"string\"}\n",
    "\n",
    "# Set the project id\n",
    "!gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🔐 Authentication\n",
    "\n",
    "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n",
    "\n",
    "- If you are using Colab to run this notebook, use the cell below and continue.\n",
    "- If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import auth\n",
    "\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save documents\n",
    "\n",
    "Save langchain documents with `SpannerDocumentSaver.add_documents(<documents>)`. To initialize `SpannerDocumentSaver` class you need to provide 3 things:\n",
    "\n",
    "1. `instance_id` - An instance of Spanner to load data from.\n",
    "1. `database_id` - An instance of Spanner database to load data from.\n",
    "1. `table_name` - The name of the table within the Spanner database to store langchain documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.documents import Document\n",
    "from langchain_google_spanner import SpannerDocumentSaver\n",
    "\n",
    "test_docs = [\n",
    "    Document(\n",
    "        page_content=\"Apple Granny Smith 150 0.99 1\",\n",
    "        metadata={\"fruit_id\": 1},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Banana Cavendish 200 0.59 0\",\n",
    "        metadata={\"fruit_id\": 2},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Orange Navel 80 1.29 1\",\n",
    "        metadata={\"fruit_id\": 3},\n",
    "    ),\n",
    "]\n",
    "\n",
    "saver = SpannerDocumentSaver(\n",
    "    instance_id=INSTANCE_ID,\n",
    "    database_id=DATABASE_ID,\n",
    "    table_name=TABLE_NAME,\n",
    ")\n",
    "saver.add_documents(test_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying for Documents from Spanner\n",
    "\n",
    "For more details on connecting to a Spanner table, please check the [Python SDK documentation](https://cloud.google.com/python/docs/reference/spanner/latest)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load documents from table\n",
    "\n",
    "Load langchain documents with `SpannerLoader.load()` or `SpannerLoader.lazy_load()`. `lazy_load` returns a generator that only queries database during the iteration. To initialize `SpannerLoader` class you need to provide:\n",
    "\n",
    "1. `instance_id` - An instance of Spanner to load data from.\n",
    "1. `database_id` - An instance of Spanner database to load data from.\n",
    "1. `query` - A query of the database dialect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_spanner import SpannerLoader\n",
    "\n",


>[Confluence](https://www.atlassian.com/software/confluence) is a wiki collaboration platform that saves and organizes all of the project-related material. `Confluence` is a knowledge base that primarily handles content management activities.

This currently supports `username/api_key`, `Oauth2 login`. Additionally, on-prem installations also support `token` authentication.

Specify a list `page_id`-s and/or `space_key` to load in the corresponding pages into Document objects, if both are specified the union of both sets will be returned.

You can also specify a boolean `include_attachments` to include attachments, this is set to False by default, if set to True all attachments will be downloaded and ConfluenceReader will extract the text from the attachments and add it to the Document object. Currently supported attachment types are: `PDF`, `PNG`, `JPEG/JPG`, `SVG`, `Word` and `Excel`.

Hint: `space_key` and `page_id` can both be found in the URL of a page in Confluence - https://yoursite.atlassian.com/wiki/spaces/<space_key>/pages/<page_id>

Before using ConfluenceLoader make sure you have the latest version of the atlassian-python-api package installed:

%pip install --upgrade --quiet  atlassian-python-api

### Username and Password or Username and API Token (Atlassian Cloud only)

This example authenticates using either a username and password or, if you're connecting to an Atlassian Cloud hosted version of Confluence, a username and an API Token.

You can generate an API token at: https://id.atlassian.com/manage-profile/security/api-tokens.

The `limit` parameter specifies how many documents will be retrieved in a single call, not how many documents will be retrieved in total.

By default the code will return up to 1000 documents in 50 documents batches. To control the total number of documents use the `max_pages` parameter.

Plese note the maximum value for the `limit` parameter in the atlassian-python-api package is currently 100.

from langchain_community.document_loaders import ConfluenceLoader

documents = loader.load(space_key="SPACE", include_attachments=True, limit=50)

### Personal Access Token (Server/On-Prem only)

This method is valid for the Data Center/Server on-prem edition only.

For more information on how to generate a Personal Access Token (PAT) check the official Confluence documentation at: https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html.

When using a PAT you provide only the token value, you cannot provide a username.

Please note that ConfluenceLoader will run under the permissions of the user that generated the PAT and will only be able to load documents for which said user has access to.

loader = ConfluenceLoader(url="https://yoursite.atlassian.com/wiki", token="12345")

space_key="SPACE", include_attachments=True, limit=50, max_pages=50


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UnstructuredPDFLoader\n",
    "\n",
    "## Overview\n",
    "\n",
    "[Unstructured](https://unstructured-io.github.io/unstructured/) supports a common interface for working with unstructured or semi-structured file formats, such as Markdown or PDF. LangChain's [UnstructuredPDFLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.UnstructuredPDFLoader.html) integrates with Unstructured to parse PDF documents into LangChain [Document](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html) objects.\n",
    "\n",
    "Please see [this page](/docs/integrations/providers/unstructured/) for more information on installing system requirements.\n",
    "\n",
    "\n",
    "### Integration details\n",
    "\n",
    "\n",
    "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/document_loaders/file_loaders/unstructured/)|\n",
    "| :--- | :--- | :---: | :---: |  :---: |\n",
    "| [UnstructuredPDFLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.UnstructuredPDFLoader.html) | [langchain_community](https://python.langchain.com/api_reference/community/index.html) | ✅ | ❌ | ✅ | \n",
    "### Loader features\n",
    "| Source | Document Lazy Loading | Native Async Support\n",
    "| :---: | :---: | :---: | \n",
    "| UnstructuredPDFLoader | ✅ | ❌ | \n",
    "\n",
    "## Setup\n",
    "\n",
    "### Credentials\n",
    "\n",
    "No credentials are needed to use this loader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to get automated best in-class tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",
    "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation\n",
    "\n",
    "Install **langchain_community** and **unstructured**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU langchain-community unstructured"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "Now we can initialize our loader:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredPDFLoader\n",
    "\n",
    "file_path = \"./example_data/layout-parser-paper.pdf\"\n",
    "loader = UnstructuredPDFLoader(file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [


>[Microsoft PowerPoint](https://en.wikipedia.org/wiki/Microsoft_PowerPoint) is a presentation program by Microsoft.

This covers how to load `Microsoft PowerPoint` documents into a document format that we can use downstream.

Please see [this guide](/docs/integrations/providers/unstructured/) for more instructions on setting up Unstructured locally, including setting up required system dependencies.

# Install packages
%pip install unstructured
%pip install python-magic
%pip install python-pptx

[Document(page_content='Adding a Bullet Slide\n\nFind the bullet slide layout\n\nUse _TextFrame.text for first bullet\n\nUse _TextFrame.add_paragraph() for subsequent bullets\n\nHere is a lot of text!\n\nHere is some text in a text box!', metadata={'source': './example_data/fake-power-point.pptx'})]

from langchain_community.document_loaders import UnstructuredPowerPointLoader

Under the hood, `Unstructured` creates different "elements" for different chunks of text. By default we combine those together, but you can easily keep that separation by specifying `mode="elements"`.

Document(page_content='Adding a Bullet Slide', metadata={'source': './example_data/fake-power-point.pptx', 'category_depth': 0, 'file_directory': './example_data', 'filename': 'fake-power-point.pptx', 'last_modified': '2023-12-19T13:42:18', 'page_number': 1, 'languages': ['eng'], 'filetype': 'application/vnd.openxmlformats-officedocument.presentationml.presentation', 'category': 'Title'})

>[Azure AI Document Intelligence](https://aka.ms/doc-intelligence) (formerly known as `Azure Form Recognizer`) is machine-learning

>based service that extracts texts (including handwriting), tables, document structures (e.g., titles, section headings, etc.) and key-value-pairs from

>digital or scanned PDFs, images, Office and HTML files.

>Document Intelligence supports `PDF`, `JPEG/JPG`, `PNG`, `BMP`, `TIFF`, `HEIF`, `DOCX`, `XLSX`, `PPTX` and `HTML`.

This current implementation of a loader using `Document Intelligence` can incorporate content page-wise and turn it into LangChain documents. The default output format is markdown, which can be easily chained with `MarkdownHeaderTextSplitter` for semantic document chunking. You can also use `mode="single"` or `mode="page"` to return pure texts in a single page or document split by page.

An Azure AI Document Intelligence resource in one of the 3 preview regions: **East US**, **West US2**, **West Europe** - follow [this document](https://learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource?view=doc-intel-4.0.0) to create one if you don't have. You will be passing `<endpoint>` and `<key>` as parameters to the loader.

%pip install --upgrade --quiet  langchain langchain-community azure-ai-documentintelligence

from langchain_community.document_loaders import AzureAIDocumentIntelligenceLoader

api_endpoint=endpoint, api_key=key, file_path=file_path, api_model="prebuilt-layout"


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSV\n",
    "\n",
    ">A [comma-separated values (CSV)](https://en.wikipedia.org/wiki/Comma-separated_values) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas.\n",
    "\n",
    "Load [csv](https://en.wikipedia.org/wiki/Comma-separated_values) data with a single row per document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(page_content='Team: Nationals\\n\"Payroll (millions)\": 81.34\\n\"Wins\": 98', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 0}), Document(page_content='Team: Reds\\n\"Payroll (millions)\": 82.20\\n\"Wins\": 97', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 1}), Document(page_content='Team: Yankees\\n\"Payroll (millions)\": 197.96\\n\"Wins\": 95', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 2}), Document(page_content='Team: Giants\\n\"Payroll (millions)\": 117.62\\n\"Wins\": 94', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 3}), Document(page_content='Team: Braves\\n\"Payroll (millions)\": 83.31\\n\"Wins\": 94', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 4}), Document(page_content='Team: Athletics\\n\"Payroll (millions)\": 55.37\\n\"Wins\": 94', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 5}), Document(page_content='Team: Rangers\\n\"Payroll (millions)\": 120.51\\n\"Wins\": 93', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 6}), Document(page_content='Team: Orioles\\n\"Payroll (millions)\": 81.43\\n\"Wins\": 93', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 7}), Document(page_content='Team: Rays\\n\"Payroll (millions)\": 64.17\\n\"Wins\": 90', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 8}), Document(page_content='Team: Angels\\n\"Payroll (millions)\": 154.49\\n\"Wins\": 89', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 9}), Document(page_content='Team: Tigers\\n\"Payroll (millions)\": 132.30\\n\"Wins\": 88', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 10}), Document(page_content='Team: Cardinals\\n\"Payroll (millions)\": 110.30\\n\"Wins\": 88', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 11}), Document(page_content='Team: Dodgers\\n\"Payroll (millions)\": 95.14\\n\"Wins\": 86', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 12}), Document(page_content='Team: White Sox\\n\"Payroll (millions)\": 96.92\\n\"Wins\": 85', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 13}), Document(page_content='Team: Brewers\\n\"Payroll (millions)\": 97.65\\n\"Wins\": 83', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 14}), Document(page_content='Team: Phillies\\n\"Payroll (millions)\": 174.54\\n\"Wins\": 81', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 15}), Document(page_content='Team: Diamondbacks\\n\"Payroll (millions)\": 74.28\\n\"Wins\": 81', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 16}), Document(page_content='Team: Pirates\\n\"Payroll (millions)\": 63.43\\n\"Wins\": 79', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 17}), Document(page_content='Team: Padres\\n\"Payroll (millions)\": 55.24\\n\"Wins\": 76', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 18}), Document(page_content='Team: Mariners\\n\"Payroll (millions)\": 81.97\\n\"Wins\": 75', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 19}), Document(page_content='Team: Mets\\n\"Payroll (millions)\": 93.35\\n\"Wins\": 74', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 20}), Document(page_content='Team: Blue Jays\\n\"Payroll (millions)\": 75.48\\n\"Wins\": 73', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 21}), Document(page_content='Team: Royals\\n\"Payroll (millions)\": 60.91\\n\"Wins\": 72', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 22}), Document(page_content='Team: Marlins\\n\"Payroll (millions)\": 118.07\\n\"Wins\": 69', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 23}), Document(page_content='Team: Red Sox\\n\"Payroll (millions)\": 173.18\\n\"Wins\": 69', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 24}), Document(page_content='Team: Indians\\n\"Payroll (millions)\": 78.43\\n\"Wins\": 68', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 25}), Document(page_content='Team: Twins\\n\"Payroll (millions)\": 94.08\\n\"Wins\": 66', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 26}), Document(page_content='Team: Rockies\\n\"Payroll (millions)\": 78.06\\n\"Wins\": 64', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 27}), Document(page_content='Team: Cubs\\n\"Payroll (millions)\": 88.19\\n\"Wins\": 61', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 28}), Document(page_content='Team: Astros\\n\"Payroll (millions)\": 60.65\\n\"Wins\": 55', metadata={'source': './example_data/mlb_teams_2012.csv', 'row': 29})]\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders.csv_loader import CSVLoader\n",
    "\n",
    "loader = CSVLoader(file_path=\"./example_data/mlb_teams_2012.csv\")\n",
    "\n",
    "data = loader.load()\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customizing the csv parsing and loading\n",
    "\n",
    "See the [csv module](https://docs.python.org/3/library/csv.html) documentation for more information of what csv args are supported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [


"[Document(page_content='Team: Nationals\\n\"Payroll (millions)\": 81.34\\n\"Wins\": 98', metadata={'source': 'Nationals', 'row': 0}), Document(page_content='Team: Reds\\n\"Payroll (millions)\": 82.20\\n\"Wins\": 97', metadata={'source': 'Reds', 'row': 1}), Document(page_content='Team: Yankees\\n\"Payroll (millions)\": 197.96\\n\"Wins\": 95', metadata={'source': 'Yankees', 'row': 2}), Document(page_content='Team: Giants\\n\"Payroll (millions)\": 117.62\\n\"Wins\": 94', metadata={'source': 'Giants', 'row': 3}), Document(page_content='Team: Braves\\n\"Payroll (millions)\": 83.31\\n\"Wins\": 94', metadata={'source': 'Braves', 'row': 4}), Document(page_content='Team: Athletics\\n\"Payroll (millions)\": 55.37\\n\"Wins\": 94', metadata={'source': 'Athletics', 'row': 5}), Document(page_content='Team: Rangers\\n\"Payroll (millions)\": 120.51\\n\"Wins\": 93', metadata={'source': 'Rangers', 'row': 6}), Document(page_content='Team: Orioles\\n\"Payroll (millions)\": 81.43\\n\"Wins\": 93', metadata={'source': 'Orioles', 'row': 7}), Document(page_content='Team: Rays\\n\"Payroll (millions)\": 64.17\\n\"Wins\": 90', metadata={'source': 'Rays', 'row': 8}), Document(page_content='Team: Angels\\n\"Payroll (millions)\": 154.49\\n\"Wins\": 89', metadata={'source': 'Angels', 'row': 9}), Document(page_content='Team: Tigers\\n\"Payroll (millions)\": 132.30\\n\"Wins\": 88', metadata={'source': 'Tigers', 'row': 10}), Document(page_content='Team: Cardinals\\n\"Payroll (millions)\": 110.30\\n\"Wins\": 88', metadata={'source': 'Cardinals', 'row': 11}), Document(page_content='Team: Dodgers\\n\"Payroll (millions)\": 95.14\\n\"Wins\": 86', metadata={'source': 'Dodgers', 'row': 12}), Document(page_content='Team: White Sox\\n\"Payroll (millions)\": 96.92\\n\"Wins\": 85', metadata={'source': 'White Sox', 'row': 13}), Document(page_content='Team: Brewers\\n\"Payroll (millions)\": 97.65\\n\"Wins\": 83', metadata={'source': 'Brewers', 'row': 14}), Document(page_content='Team: Phillies\\n\"Payroll (millions)\": 174.54\\n\"Wins\": 81', metadata={'source': 'Phillies', 'row': 15}), Document(page_content='Team: Diamondbacks\\n\"Payroll (millions)\": 74.28\\n\"Wins\": 81', metadata={'source': 'Diamondbacks', 'row': 16}), Document(page_content='Team: Pirates\\n\"Payroll (millions)\": 63.43\\n\"Wins\": 79', metadata={'source': 'Pirates', 'row': 17}), Document(page_content='Team: Padres\\n\"Payroll (millions)\": 55.24\\n\"Wins\": 76', metadata={'source': 'Padres', 'row': 18}), Document(page_content='Team: Mariners\\n\"Payroll (millions)\": 81.97\\n\"Wins\": 75', metadata={'source': 'Mariners', 'row': 19}), Document(page_content='Team: Mets\\n\"Payroll (millions)\": 93.35\\n\"Wins\": 74', metadata={'source': 'Mets', 'row': 20}), Document(page_content='Team: Blue Jays\\n\"Payroll (millions)\": 75.48\\n\"Wins\": 73', metadata={'source': 'Blue Jays', 'row': 21}), Document(page_content='Team: Royals\\n\"Payroll (millions)\": 60.91\\n\"Wins\": 72', metadata={'source': 'Royals', 'row': 22}), Document(page_content='Team: Marlins\\n\"Payroll (millions)\": 118.07\\n\"Wins\": 69', metadata={'source': 'Marlins', 'row': 23}), Document(page_content='Team: Red Sox\\n\"Payroll (millions)\": 173.18\\n\"Wins\": 69', metadata={'source': 'Red Sox', 'row': 24}), Document(page_content='Team: Indians\\n\"Payroll (millions)\": 78.43\\n\"Wins\": 68', metadata={'source': 'Indians', 'row': 25}), Document(page_content='Team: Twins\\n\"Payroll (millions)\": 94.08\\n\"Wins\": 66', metadata={'source': 'Twins', 'row': 26}), Document(page_content='Team: Rockies\\n\"Payroll (millions)\": 78.06\\n\"Wins\": 64', metadata={'source': 'Rockies', 'row': 27}), Document(page_content='Team: Cubs\\n\"Payroll (millions)\": 88.19\\n\"Wins\": 61', metadata={'source': 'Cubs', 'row': 28}), Document(page_content='Team: Astros\\n\"Payroll (millions)\": 60.65\\n\"Wins\": 55', metadata={'source': 'Astros', 'row': 29})]\n"
     ]
    }
   ],
   "source": [
    "loader = CSVLoader(file_path=\"./example_data/mlb_teams_2012.csv\", source_column=\"Team\")\n",
    "\n",
    "data = loader.load()\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `UnstructuredCSVLoader`\n",
    "\n",
    "You can also load the table using the `UnstructuredCSVLoader`. One advantage of using `UnstructuredCSVLoader` is that if you use it in `\"elements\"` mode, an HTML representation of the table will be available in the metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "
\n",
      "
\n",
      "
\n",
      "      Team	\n",
      "      \"Payroll (millions)\"	\n",
      "      \"Wins\"	\n",
      "
\n",
      "
\n",
      "      Nationals	\n",
      "      81.34	\n",
      "      98	\n",
      "
\n",
      "
\n",
      "      Reds	\n",
      "      82.20	\n",
      "      97	\n",
      "
\n",
      "
\n",
      "      Yankees	\n",
      "      197.96	\n",
      "      95	\n",
      "
\n",
      "
\n",
      "      Giants	\n",
      "      117.62	\n",
      "      94	\n",
      "
\n",
      "
\n",
      "      Braves	\n",
      "      83.31	\n",
      "      94	\n",
      "
\n",
      "
\n",
      "      Athletics	\n",
      "      55.37	\n",
      "      94	\n",
      "
\n",
      "
\n",
      "      Rangers	\n",
      "      120.51	\n",
      "      93	\n",
      "
\n",
      "
\n",
      "      Orioles	\n",
      "      81.43	\n",
      "      93	\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google Bigtable\n",
    "\n",
    "> [Bigtable](https://cloud.google.com/bigtable) is a key-value and wide-column store, ideal for fast access to structured, semi-structured, or unstructured data. Extend your database application to build AI-powered experiences leveraging Bigtable's Langchain integrations.\n",
    "\n",
    "This notebook goes over how to use [Bigtable](https://cloud.google.com/bigtable) to [save, load and delete langchain documents](/docs/how_to#document-loaders) with `BigtableLoader` and `BigtableSaver`.\n",
    "\n",
    "Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-bigtable-python/).\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-bigtable-python/blob/main/docs/document_loader.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before You Begin\n",
    "\n",
    "To run this notebook, you will need to do the following:\n",
    "\n",
    "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
    "* [Enable the Bigtable API](https://console.cloud.google.com/flows/enableapi?apiid=bigtable.googleapis.com)\n",
    "* [Create a Bigtable instance](https://cloud.google.com/bigtable/docs/creating-instance)\n",
    "* [Create a Bigtable table](https://cloud.google.com/bigtable/docs/managing-tables)\n",
    "* [Create Bigtable access credentials](https://developers.google.com/workspace/guides/create-credentials)\n",
    "\n",
    "After confirmed access to database in the runtime environment of this notebook, filling the following values and run the cell before running example scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please specify an instance and a table for demo purpose.\n",
    "INSTANCE_ID = \"my_instance\"  # @param {type:\"string\"}\n",
    "TABLE_ID = \"my_table\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦜🔗 Library Installation\n",
    "\n",
    "The integration lives in its own `langchain-google-bigtable` package, so we need to install it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -upgrade --quiet langchain-google-bigtable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only**: Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☁ Set Your Google Cloud Project\n",
    "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n",
    "\n",
    "If you don't know your project ID, try the following:\n",
    "\n",
    "* Run `gcloud config list`.\n",
    "* Run `gcloud projects list`.\n",
    "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n",
    "\n",
    "PROJECT_ID = \"my-project-id\"  # @param {type:\"string\"}\n",
    "\n",
    "# Set the project id\n",
    "!gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🔐 Authentication\n",
    "\n",
    "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n",
    "\n",
    "- If you are using Colab to run this notebook, use the cell below and continue.\n",
    "- If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import auth\n",
    "\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the saver\n",
    "\n",
    "Save langchain documents with `BigtableSaver.add_documents(<documents>)`. To initialize `BigtableSaver` class you need to provide 2 things:\n",
    "\n",
    "1. `instance_id` - An instance of Bigtable.\n",
    "1. `table_id` - The name of the table within the Bigtable to store langchain documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.documents import Document\n",
    "from langchain_google_bigtable import BigtableSaver\n",
    "\n",
    "test_docs = [\n",
    "    Document(\n",
    "        page_content=\"Apple Granny Smith 150 0.99 1\",\n",
    "        metadata={\"fruit_id\": 1},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Banana Cavendish 200 0.59 0\",\n",
    "        metadata={\"fruit_id\": 2},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Orange Navel 80 1.29 1\",\n",
    "        metadata={\"fruit_id\": 3},\n",
    "    ),\n",
    "]\n",
    "\n",
    "saver = BigtableSaver(\n",
    "    instance_id=INSTANCE_ID,\n",
    "    table_id=TABLE_ID,\n",
    ")\n",
    "\n",
    "saver.add_documents(test_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying for Documents from Bigtable\n",
    "For more details on connecting to a Bigtable table, please check the [Python SDK documentation](https://cloud.google.com/python/docs/reference/bigtable/latest/client)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load documents from table\n",
    "\n",
    "Load langchain documents with `BigtableLoader.load()` or `BigtableLoader.lazy_load()`. `lazy_load` returns a generator that only queries database during the iteration. To initialize `BigtableLoader` class you need to provide:\n",
    "\n",
    "1. `instance_id` - An instance of Bigtable.\n",
    "1. `table_id` - The name of the table within the Bigtable to store langchain documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_bigtable import BigtableLoader\n",
    "\n",
    "loader = BigtableLoader(\n",
    "    instance_id=INSTANCE_ID,\n",
    "    table_id=TABLE_ID,\n",
    ")\n",
    "\n",
    "for doc in loader.lazy_load():\n",
    "    print(doc)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete documents\n",
    "\n",


"Use only the provided relationship types and properties in the schema.\n",
    "Do not use any other relationship types or properties that are not provided.\n",
    "Schema:\n",
    "{schema}\n",
    "Note: Do not include any explanations or apologies in your responses.\n",
    "Do not respond to any questions that might ask anything else than for you to construct a Cypher statement.\n",
    "Do not include any text except the generated Cypher statement.\n",
    "Examples: Here are a few examples of generated Cypher statements for particular questions:\n",
    "# How many people played in Top Gun?\n",
    "MATCH (m:Movie {{name:\"Top Gun\"}})<-[:ACTED_IN]-()\n",
    "RETURN count(*) AS numberOfActors\n",
    "\n",
    "The question is:\n",
    "{question}\"\"\"\n",
    "\n",
    "CYPHER_GENERATION_PROMPT = PromptTemplate(\n",
    "    input_variables=[\"schema\", \"question\"], template=CYPHER_GENERATION_TEMPLATE\n",
    ")\n",
    "\n",
    "chain = GraphCypherQAChain.from_llm(\n",
    "    ChatOpenAI(temperature=0),\n",
    "    graph=graph,\n",
    "    verbose=True,\n",
    "    cypher_prompt=CYPHER_GENERATION_PROMPT,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "47c64027-cf42-493a-9c76-2d10ba753728",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new GraphCypherQAChain chain...\u001b[0m\n",
      "Generated Cypher:\n",
      "\u001b[32;1m\u001b[1;3mMATCH (m:Movie {name:\"Top Gun\"})<-[:ACTED_IN]-()\n",
      "RETURN count(*) AS numberOfActors\u001b[0m\n",
      "Full Context:\n",
      "\u001b[32;1m\u001b[1;3m[{'numberOfActors': 4}]\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'How many people played in Top Gun?',\n",
       " 'result': 'There were 4 actors in Top Gun.'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"query\": \"How many people played in Top Gun?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e721cad-aa87-4526-9231-2dfc0e365939",
   "metadata": {},
   "source": [
    "## Use separate LLMs for Cypher and answer generation\n",
    "You can use the `cypher_llm` and `qa_llm` parameters to define different llms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6f9becc2-f579-45bf-9b50-2ce02bde92da",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = GraphCypherQAChain.from_llm(\n",
    "    graph=graph,\n",
    "    cypher_llm=ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo\"),\n",
    "    qa_llm=ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-16k\"),\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ff18e3e3-3402-4683-aec4-a19898f23ca1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new GraphCypherQAChain chain...\u001b[0m\n",
      "Generated Cypher:\n",
      "\u001b[32;1m\u001b[1;3mMATCH (a:Actor)-[:ACTED_IN]->(m:Movie)\n",
      "WHERE m.name = 'Top Gun'\n",
      "RETURN a.name\u001b[0m\n",
      "Full Context:\n",
      "\u001b[32;1m\u001b[1;3m[{'a.name': 'Tom Cruise'}, {'a.name': 'Val Kilmer'}, {'a.name': 'Anthony Edwards'}, {'a.name': 'Meg Ryan'}]\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'Who played in Top Gun?',\n",
       " 'result': 'Tom Cruise, Val Kilmer, Anthony Edwards, and Meg Ryan played in Top Gun.'}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"query\": \"Who played in Top Gun?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eefea16b-508f-4552-8942-9d5063ed7d37",
   "metadata": {},
   "source": [
    "## Ignore specified node and relationship types\n",
    "\n",
    "You can use `include_types` or `exclude_types` to ignore parts of the graph schema when generating Cypher statements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a20fa21e-fb85-41c4-aac0-53fb25e34604",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = GraphCypherQAChain.from_llm(\n",
    "    graph=graph,\n",
    "    cypher_llm=ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo\"),\n",
    "    qa_llm=ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-16k\"),\n",
    "    verbose=True,\n",
    "    exclude_types=[\"Movie\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3ad7f6b8-543e-46e4-a3b2-40fa3e66e895",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node properties are the following:\n",
      "Actor {name: STRING}\n",
      "Relationship properties are the following:\n",
      "\n",
      "The relationships are the following:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect graph schema\n",
    "print(chain.graph_schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0202e88-d700-40ed-aef9-0c969c7bf951",
   "metadata": {},
   "source": [
    "## Validate generated Cypher statements\n",
    "You can use the `validate_cypher` parameter to validate and correct relationship directions in generated Cypher statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "53665d03-7afd-433c-bdd5-750127bfb152",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = GraphCypherQAChain.from_llm(\n",
    "    llm=ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo\"),\n",
    "    graph=graph,\n",
    "    verbose=True,\n",
    "    validate_cypher=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "19e1a591-9c10-4d7b-aa36-a5e1b778a97b",
   "metadata": {},
   "outputs": [
    {


"Node name: 'Publisher', Node properties: [{'property': 'name', 'type': 'str'}]\n",
    "\n",
    "Relationship properties are the following:\n",
    "\n",
    "The relationships are the following:\n",
    "['(:Game)-[:AVAILABLE_ON]->(:Platform)']\n",
    "['(:Game)-[:HAS_GENRE]->(:Genre)']\n",
    "['(:Game)-[:PUBLISHED_BY]->(:Publisher)']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d3a1da",
   "metadata": {},
   "source": [
    "## Querying the database"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8aedfd63",
   "metadata": {},
   "source": [
    "To interact with the OpenAI API, you must configure your API key as an environment variable using the Python [os](https://docs.python.org/3/library/os.html) package. This ensures proper authorization for your requests. You can find more information on obtaining your API key [here](https://help.openai.com/en/articles/4936850-where-do-i-find-my-secret-api-key)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8385c72",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OPENAI_API_KEY\"] = \"your-key-here\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a74565a",
   "metadata": {},
   "source": [
    "You should create the graph chain using the following script, which will be utilized in the question-answering process based on your graph data. While it defaults to GPT-3.5-turbo, you might also consider experimenting with other models like [GPT-4](https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4) for notably improved Cypher queries and outcomes. We'll utilize the OpenAI chat, utilizing the key you previously configured. We'll set the temperature to zero, ensuring predictable and consistent answers. Additionally, we'll use our Memgraph-LangChain graph and set the verbose parameter, which defaults to False, to True to receive more detailed messages regarding query generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a3a5f2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = GraphCypherQAChain.from_llm(\n",
    "    ChatOpenAI(temperature=0), graph=graph, verbose=True, model_name=\"gpt-3.5-turbo\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "949de4f3",
   "metadata": {},
   "source": [
    "Now you can start asking questions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7aea263",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = chain.run(\"Which platforms is Baldur's Gate 3 available on?\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a06a8164",
   "metadata": {},
   "source": [
    "```\n",
    "> Entering new GraphCypherQAChain chain...\n",
    "Generated Cypher:\n",
    "MATCH (g:Game {name: 'Baldur\\'s Gate 3'})-[:AVAILABLE_ON]->(p:Platform)\n",
    "RETURN p.name\n",
    "Full Context:\n",
    "[{'p.name': 'PlayStation 5'}, {'p.name': 'Mac OS'}, {'p.name': 'Windows'}, {'p.name': 'Xbox Series X/S'}]\n",
    "\n",
    "> Finished chain.\n",
    "Baldur's Gate 3 is available on PlayStation 5, Mac OS, Windows, and Xbox Series X/S.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d298d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = chain.run(\"Is Baldur's Gate 3 available on Windows?\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99dd783c",
   "metadata": {},
   "source": [
    "```\n",
    "> Entering new GraphCypherQAChain chain...\n",
    "Generated Cypher:\n",
    "MATCH (:Game {name: 'Baldur\\'s Gate 3'})-[:AVAILABLE_ON]->(:Platform {name: 'Windows'})\n",
    "RETURN true\n",
    "Full Context:\n",
    "[{'true': True}]\n",
    "\n",
    "> Finished chain.\n",
    "Yes, Baldur's Gate 3 is available on Windows.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08620465",
   "metadata": {},
   "source": [
    "## Chain modifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6603e6c8",
   "metadata": {},
   "source": [
    "To modify the behavior of your chain and obtain more context or additional information, you can modify the chain's parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d187a83",
   "metadata": {},
   "source": [
    "#### Return direct query results\n",
    "The `return_direct` modifier specifies whether to return the direct results of the executed Cypher query or the processed natural language response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0533847d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return the result of querying the graph directly\n",
    "chain = GraphCypherQAChain.from_llm(\n",
    "    ChatOpenAI(temperature=0), graph=graph, verbose=True, return_direct=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afbe96fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = chain.run(\"Which studio published Baldur's Gate 3?\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94b32b6e",
   "metadata": {},
   "source": [
    "```\n",
    "> Entering new GraphCypherQAChain chain...\n",
    "Generated Cypher:\n",
    "MATCH (:Game {name: 'Baldur\\'s Gate 3'})-[:PUBLISHED_BY]->(p:Publisher)\n",
    "RETURN p.name\n",
    "\n",
    "> Finished chain.\n",
    "[{'p.name': 'Larian Studios'}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c97ab3a",
   "metadata": {},
   "source": [
    "#### Return query intermediate steps\n",
    "The `return_intermediate_steps` chain modifier enhances the returned response by including the intermediate steps of the query in addition to the initial query result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82f673c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Return all the intermediate steps of query execution\n",
    "chain = GraphCypherQAChain.from_llm(\n",
    "    ChatOpenAI(temperature=0), graph=graph, verbose=True, return_intermediate_steps=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d87e0976",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = chain(\"Is Baldur's Gate 3 an Adventure game?\")\n",
    "print(f\"Intermediate steps: {response['intermediate_steps']}\")\n",
    "print(f\"Final response: {response['result']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df12b3da",
   "metadata": {},
   "source": [
    "```\n",
    "> Entering new GraphCypherQAChain chain...\n",
    "Generated Cypher:\n",
    "MATCH (g:Game {name: 'Baldur\\'s Gate 3'})-[:HAS_GENRE]->(genre:Genre {name: 'Adventure'})\n",


{
 "cells": [
  {
   "cell_type": "raw",
   "id": "675d11f1",
   "metadata": {},
   "source": [
    "---\n",
    "keywords: [gemini, GoogleGenerativeAI, gemini-pro]\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aZWXpbf0Eph",
   "metadata": {
    "id": "7aZWXpbf0Eph"
   },
   "source": [
    "# Google AI\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bead5ede-d9cc-44b9-b062-99c90a10cf40",
   "metadata": {},
   "source": [
    ":::caution\n",
    "You are currently on a page documenting the use of Google models as [text completion models](/docs/concepts/#llms). Many popular Google models are [chat completion models](/docs/concepts/#chat-models).\n",
    "\n",
    "You may be looking for [this page instead](/docs/integrations/chat/google_generative_ai/).\n",
    ":::\n",
    "\n",
    "A guide on using [Google Generative AI](https://developers.generativeai.google/) models with Langchain. Note: It's separate from Google Cloud Vertex AI [integration](/docs/integrations/llms/google_vertex_ai_palm)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "H4AjsqTswBCE",
   "metadata": {
    "id": "H4AjsqTswBCE"
   },
   "source": [
    "## Setting up\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "EFHNUieMwJrl",
   "metadata": {
    "id": "EFHNUieMwJrl"
   },
   "source": [
    "To use Google Generative AI you must install the `langchain-google-genai` Python package and generate an API key. [Read more details](https://developers.generativeai.google/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8Qzm6SqKwgak",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain-google-genai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ONb7ZtOwjbo",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_genai import GoogleGenerativeAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "X3pjCW0i22gm",
   "metadata": {},
   "outputs": [],
   "source": [
    "from getpass import getpass\n",
    "\n",
    "api_key = getpass()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "GT50LgFP0j-w",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Pros of Python:**\n",
      "\n",
      "* **Easy to learn:** Python is a very easy-to-learn programming language, even for beginners. Its syntax is simple and straightforward, and there are a lot of resources available to help you get started.\n",
      "* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, and machine learning. It's also a good choice for beginners because it can be used for a variety of projects, so you can learn the basics and then move on to more complex tasks.\n",
      "* **High-level:** Python is a high-level programming language, which means that it's closer to human language than other programming languages. This makes it easier to read and understand, which can be a big advantage for beginners.\n",
      "* **Open-source:** Python is an open-source programming language, which means that it's free to use and there are a lot of resources available to help you learn it.\n",
      "* **Community:** Python has a large and active community of developers, which means that there are a lot of people who can help you if you get stuck.\n",
      "\n",
      "**Cons of Python:**\n",
      "\n",
      "* **Slow:** Python is a relatively slow programming language compared to some other languages, such as C++. This can be a disadvantage if you're working on computationally intensive tasks.\n",
      "* **Not as performant:** Python is not as performant as some other programming languages, such as C++ or Java. This can be a disadvantage if you're working on projects that require high performance.\n",
      "* **Dynamic typing:** Python is a dynamically typed programming language, which means that the type of a variable can change during runtime. This can be a disadvantage if you need to ensure that your code is type-safe.\n",
      "* **Unmanaged memory:** Python uses a garbage collection system to manage memory. This can be a disadvantage if you need to have more control over memory management.\n",
      "\n",
      "Overall, Python is a very good programming language for beginners. It's easy to learn, versatile, and has a large community of developers. However, it's important to be aware of its limitations, such as its slow performance and lack of performance.\n"
     ]
    }
   ],
   "source": [
    "llm = GoogleGenerativeAI(model=\"models/text-bison-001\", google_api_key=api_key)\n",
    "print(\n",
    "    llm.invoke(\n",
    "        \"What are some of the pros and cons of Python as a programming language?\"\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "TSGdxkJtwl8-",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Pros:**\n",
      "\n",
      "* **Simplicity and Readability:** Python is known for its simple and easy-to-read syntax, which makes it accessible to beginners and reduces the chance of errors. It uses indentation to define blocks of code, making the code structure clear and visually appealing.\n",
      "\n",
      "* **Versatility:** Python is a general-purpose language, meaning it can be used for a wide range of tasks, including web development, data science, machine learning, and desktop applications. This versatility makes it a popular choice for various projects and industries.\n",
      "\n",
      "* **Large Community:** Python has a vast and active community of developers, which contributes to its growth and popularity. This community provides extensive documentation, tutorials, and open-source libraries, making it easy for Python developers to find support and resources.\n",
      "\n",
      "* **Extensive Libraries:** Python offers a rich collection of libraries and frameworks for various tasks, such as data analysis (NumPy, Pandas), web development (Django, Flask), machine learning (Scikit-learn, TensorFlow), and many more. These libraries provide pre-built functions and modules, allowing developers to quickly and efficiently solve common problems.\n",
      "\n",
      "* **Cross-Platform Support:** Python is cross-platform, meaning it can run on various operating systems, including Windows, macOS, and Linux. This allows developers to write code that can be easily shared and used across different platforms.\n",
      "\n",
      "**Cons:**\n",
      "\n",
      "* **Speed and Performance:** Python is generally slower than compiled languages like C++ or Java due to its interpreted nature. This can be a disadvantage for performance-intensive tasks, such as real-time systems or heavy numerical computations.\n",
      "\n",
      "* **Memory Usage:** Python programs tend to consume more memory compared to compiled languages. This is because Python uses a dynamic memory allocation system, which can lead to memory fragmentation and higher memory usage.\n",
      "\n",
      "* **Lack of Static Typing:** Python is a dynamically typed language, which means that data types are not explicitly defined for variables. This can make it challenging to detect type errors during development, which can lead to unexpected behavior or errors at runtime.\n",
      "\n",
      "* **GIL (Global Interpreter Lock):** Python uses a global interpreter lock (GIL) to ensure that only one thread can execute Python bytecode at a time. This can limit the scalability and parallelism of Python programs, especially in multi-threaded or multiprocessing scenarios.\n",
      "\n",
      "* **Package Management:** While Python has a vast ecosystem of libraries and packages, managing dependencies and package versions can be challenging. The Python Package Index (PyPI) is the official repository for Python packages, but it can be difficult to ensure compatibility and avoid conflicts between different versions of packages.\n"
     ]
    }
   ],
   "source": [
    "llm = GoogleGenerativeAI(model=\"gemini-pro\", google_api_key=api_key)\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9e9b7651",
   "metadata": {},
   "source": [
    "# Azure OpenAI\n",
    "\n",
    ":::caution\n",
    "You are currently on a page documenting the use of Azure OpenAI [text completion models](/docs/concepts/#llms). The latest and most popular Azure OpenAI models are [chat completion models](/docs/concepts/#chat-models).\n",
    "\n",
    "Unless you are specifically using `gpt-3.5-turbo-instruct`, you are probably looking for [this page instead](/docs/integrations/chat/azure_chat_openai/).\n",
    ":::\n",
    "\n",
    "This page goes over how to use LangChain with [Azure OpenAI](https://aka.ms/azure-openai).\n",
    "\n",
    "The Azure OpenAI API is compatible with OpenAI's API.  The `openai` Python package makes it easy to use both OpenAI and Azure OpenAI.  You can call Azure OpenAI the same way you call OpenAI with the exceptions noted below.\n",
    "\n",
    "## API configuration\n",
    "You can configure the `openai` package to use Azure OpenAI using environment variables.  The following is for `bash`:\n",
    "\n",
    "```bash\n",
    "# The API version you want to use: set this to `2023-12-01-preview` for the released version.\n",
    "export OPENAI_API_VERSION=2023-12-01-preview\n",
    "# The base URL for your Azure OpenAI resource.  You can find this in the Azure portal under your Azure OpenAI resource.\n",
    "export AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com\n",
    "# The API key for your Azure OpenAI resource.  You can find this in the Azure portal under your Azure OpenAI resource.\n",
    "export AZURE_OPENAI_API_KEY=<your Azure OpenAI API key>\n",
    "```\n",
    "\n",
    "Alternatively, you can configure the API right within your running Python environment:\n",
    "\n",
    "```python\n",
    "import os\n",
    "os.environ[\"OPENAI_API_VERSION\"] = \"2023-12-01-preview\"\n",
    "```\n",
    "\n",
    "## Azure Active Directory Authentication\n",
    "There are two ways you can authenticate to Azure OpenAI:\n",
    "- API Key\n",
    "- Azure Active Directory (AAD)\n",
    "\n",
    "Using the API key is the easiest way to get started. You can find your API key in the Azure portal under your Azure OpenAI resource.\n",
    "\n",
    "However, if you have complex security requirements - you may want to use Azure Active Directory. You can find more information on how to use AAD with Azure OpenAI [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity).\n",
    "\n",
    "If you are developing locally, you will need to have the Azure CLI installed and be logged in. You can install the Azure CLI [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli). Then, run `az login` to log in.\n",
    "\n",
    "Add a role an Azure role assignment `Cognitive Services OpenAI User` scoped to your Azure OpenAI resource. This will allow you to get a token from AAD to use with Azure OpenAI. You can grant this role assignment to a user, group, service principal, or managed identity. For more information about Azure OpenAI RBAC roles see [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/role-based-access-control).\n",
    "\n",
    "To use AAD in Python with LangChain, install the `azure-identity` package. Then, set `OPENAI_API_TYPE` to `azure_ad`. Next, use the `DefaultAzureCredential` class to get a token from AAD by calling `get_token` as shown below. Finally, set the `OPENAI_API_KEY` environment variable to the token value.\n",
    "\n",
    "```python\n",
    "import os\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# Get the Azure Credential\n",
    "credential = DefaultAzureCredential()\n",
    "\n",
    "# Set the API type to `azure_ad`\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure_ad\"\n",
    "# Set the API_KEY to the token from the Azure credential\n",
    "os.environ[\"OPENAI_API_KEY\"] = credential.get_token(\"https://cognitiveservices.azure.com/.default\").token\n",
    "```\n",
    "\n",
    "The `DefaultAzureCredential` class is an easy way to get started with AAD authentication. You can also customize the credential chain if necessary. In the example shown below, we first try Managed Identity, then fall back to the Azure CLI. This is useful if you are running your code in Azure, but want to develop locally.\n",
    "\n",
    "```python\n",
    "from azure.identity import ChainedTokenCredential, ManagedIdentityCredential, AzureCliCredential\n",
    "\n",
    "credential = ChainedTokenCredential(\n",
    "    ManagedIdentityCredential(),\n",
    "    AzureCliCredential()\n",
    ")\n",
    "```\n",
    "\n",
    "## Deployments\n",
    "With Azure OpenAI, you set up your own deployments of the common GPT-3 and Codex models.  When calling the API, you need to specify the deployment you want to use.\n",
    "\n",
    "_**Note**: These docs are for the Azure text completion models. Models like GPT-4 are chat models. They have a slightly different interface, and can be accessed via the `AzureChatOpenAI` class. For docs on Azure chat see [Azure Chat OpenAI documentation](/docs/integrations/chat/azure_chat_openai)._\n",
    "\n",
    "Let's say your deployment name is `gpt-35-turbo-instruct-prod`.  In the `openai` Python API, you can specify this deployment with the `engine` parameter.  For example:\n",
    "\n",
    "```python\n",
    "import openai\n",
    "\n",
    "client = AzureOpenAI(\n",
    "    api_version=\"2023-12-01-preview\",\n",
    ")\n",
    "\n",
    "response = client.completions.create(\n",
    "    model=\"gpt-35-turbo-instruct-prod\",\n",
    "    prompt=\"Test prompt\"\n",
    ")\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89fdb593-5a42-4098-87b7-1496fa511b1c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "faacfa54",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_VERSION\"] = \"2023-12-01-preview\"\n",
    "os.environ[\"AZURE_OPENAI_ENDPOINT\"] = \"...\"\n",
    "os.environ[\"AZURE_OPENAI_API_KEY\"] = \"...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8fad2a6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Azure OpenAI\n",
    "from langchain_openai import AzureOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8c80213a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of Azure OpenAI\n",
    "# Replace the deployment name with your own\n",
    "llm = AzureOpenAI(\n",
    "    deployment_name=\"gpt-35-turbo-instruct-0914\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "592dc404",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\" Why couldn't the bicycle stand up by itself?\\n\\nBecause it was two-tired!\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run the LLM\n",
    "llm.invoke(\"Tell me a joke\")"
   ]
  },
  {


"cell_type": "markdown",
   "id": "bbfebea1",
   "metadata": {},
   "source": [
    "We can also print the LLM and see its custom print."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9c33fa19",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mAzureOpenAI\u001b[0m\n",
      "Params: {'deployment_name': 'gpt-35-turbo-instruct-0914', 'model_name': 'gpt-3.5-turbo-instruct', 'temperature': 0.7, 'top_p': 1, 'frequency_penalty': 0, 'presence_penalty': 0, 'n': 1, 'logit_bias': {}, 'max_tokens': 256}\n"
     ]
    }
   ],
   "source": [
    "print(llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a8b5917",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "3bae61d45a4f4d73ecea8149862d4bfbae7d4d4a2f71b6e609a1be8f6c8d4298"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}