"id": "5f298bc0",
   "metadata": {},
   "source": [
    "Any data added through `langchain-weaviate` will persist in Weaviate according to its configuration. \n",
    "\n",
    "WCS instances, for example, are configured to persist data indefinitely, and Docker instances can be set up to persist data in a volume. Read more about [Weaviate's persistence](https://weaviate.io/developers/weaviate/configuration/persistence)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da874a61",
   "metadata": {},
   "source": [
    "## Multi-tenancy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67a0719f",
   "metadata": {},
   "source": [
    "[Multi-tenancy](https://weaviate.io/developers/weaviate/concepts/data#multi-tenancy) allows you to have a high number of isolated collections of data, with the same collection configuration, in a single Weaviate instance. This is great for multi-user environments such as building a SaaS app, where each end user will have their own isolated data collection.\n",
    "\n",
    "To use multi-tenancy, the vector store need to be aware of the `tenant` parameter. \n",
    "\n",
    "So when adding any data, provide the `tenant` parameter as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8d365855",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-Mar-26 03:40 PM - langchain_weaviate.vectorstores - INFO - Tenant Foo does not exist in index LangChain_30b9273d43b3492db4fb2aba2e0d6871. Creating tenant.\n"
     ]
    }
   ],
   "source": [
    "db_with_mt = WeaviateVectorStore.from_documents(\n",
    "    docs, embeddings, client=weaviate_client, tenant=\"Foo\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b3e6107",
   "metadata": {},
   "source": [
    "And when performing queries, provide the `tenant` parameter also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "49659eb3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', metadata={'source': 'state_of_the_union.txt'}),\n",
       " Document(page_content='And so many families are living paycheck to paycheck, struggling to keep up with the rising cost of food, gas, housing, and so much more. \\n\\nI understand. \\n\\nI remember when my Dad had to leave our home in Scranton, Pennsylvania to find work. I grew up in a family where if the price of food went up, you felt it. \\n\\nThat’s why one of the first things I did as President was fight to pass the American Rescue Plan.  \\n\\nBecause people were hurting. We needed to act, and we did. \\n\\nFew pieces of legislation have done more in a critical moment in our history to lift us out of crisis. \\n\\nIt fueled our efforts to vaccinate the nation and combat COVID-19. It delivered immediate economic relief for tens of millions of Americans.  \\n\\nHelped put food on their table, keep a roof over their heads, and cut the cost of health insurance. \\n\\nAnd as my Dad used to say, it gave people a little breathing room.', metadata={'source': 'state_of_the_union.txt'}),\n",
       " Document(page_content='He and his Dad both have Type 1 diabetes, which means they need insulin every day. Insulin costs about $10 a vial to make.  \\n\\nBut drug companies charge families like Joshua and his Dad up to 30 times more. I spoke with Joshua’s mom. \\n\\nImagine what it’s like to look at your child who needs insulin and have no idea how you’re going to pay for it.  \\n\\nWhat it does to your dignity, your ability to look your child in the eye, to be the parent you expect to be. \\n\\nJoshua is here with us tonight. Yesterday was his birthday. Happy birthday, buddy.  \\n\\nFor Joshua, and for the 200,000 other young people with Type 1 diabetes, let’s cap the cost of insulin at $35 a month so everyone can afford it.  \\n\\nDrug companies will still do very well. And while we’re at it let Medicare negotiate lower prices for prescription drugs, like the VA already does.', metadata={'source': 'state_of_the_union.txt'}),\n",
       " Document(page_content='Putin’s latest attack on Ukraine was premeditated and unprovoked. \\n\\nHe rejected repeated efforts at diplomacy. \\n\\nHe thought the West and NATO wouldn’t respond. And he thought he could divide us at home. Putin was wrong. We were ready.  Here is what we did.   \\n\\nWe prepared extensively and carefully. \\n\\nWe spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin. \\n\\nI spent countless hours unifying our European allies. We shared with the world in advance what we knew Putin was planning and precisely how he would try to falsely justify his aggression.  \\n\\nWe countered Russia’s lies with truth.   \\n\\nAnd now that he has acted the free world is holding him accountable. \\n\\nAlong with twenty-seven members of the European Union including France, Germany, Italy, as well as countries like the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.', metadata={'source': 'state_of_the_union.txt'})]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_with_mt.similarity_search(query, tenant=\"Foo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ecf858",
   "metadata": {},
   "source": [
    "## Retriever options"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e3757a",
   "metadata": {},
   "source": [
    "Weaviate can also be used as a retriever"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2a8712d",
   "metadata": {},
   "source": [
    "### Maximal marginal relevance search (MMR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c92add51",
   "metadata": {},
   "source": [
    "In addition to using similaritysearch  in the retriever object, you can also use `mmr`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cb302651",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/workspaces/langchain-weaviate/.venv/lib/python3.12/site-packages/pydantic/main.py:1024: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/\n",
      "  warnings.warn('The `dict` method is deprecated; use `model_dump` instead.', category=PydanticDeprecatedSince20)\n"
     ]
    },
    {
     "data": {
      "text/plain": [


"/workspaces/langchain-weaviate/.venv/lib/python3.12/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.\n",
      "  warn_deprecated(\n",
      "/workspaces/langchain-weaviate/.venv/lib/python3.12/site-packages/pydantic/main.py:1024: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/\n",
      "  warnings.warn('The `dict` method is deprecated; use `model_dump` instead.', category=PydanticDeprecatedSince20)\n",
      "/workspaces/langchain-weaviate/.venv/lib/python3.12/site-packages/pydantic/main.py:1024: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/\n",
      "  warnings.warn('The `dict` method is deprecated; use `model_dump` instead.', category=PydanticDeprecatedSince20)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'answer': ' The president thanked Justice Stephen Breyer for his service and announced his nomination of Judge Ketanji Brown Jackson to the Supreme Court.\\n',\n",
       " 'sources': '31-pl'}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain(\n",
    "    {\"question\": \"What did the president say about Justice Breyer\"},\n",
    "    return_only_outputs=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fecab3b5",
   "metadata": {},
   "source": [
    "### Retrieval-Augmented Generation\n",
    "\n",
    "Another very popular application of combining LLMs and vector stores is retrieval-augmented generation (RAG). This is a technique that uses a retriever to find relevant information from a vector store, and then uses an LLM to provide an output based on the retrieved data and a prompt.\n",
    "\n",
    "We begin with a similar setup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "33b0a9d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "texts = text_splitter.split_text(state_of_the_union)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d2ade6ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "docsearch = WeaviateVectorStore.from_texts(\n",
    "    texts,\n",
    "    embeddings,\n",
    "    client=weaviate_client,\n",
    "    metadatas=[{\"source\": f\"{i}-pl\"} for i in range(len(texts))],\n",
    ")\n",
    "\n",
    "retriever = docsearch.as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39413671",
   "metadata": {},
   "source": [
    "We need to construct a template for the RAG model so that the retrieved information will be populated in the template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "578570b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_variables=['context', 'question'] messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'question'], template=\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\\nQuestion: {question}\\nContext: {context}\\nAnswer:\\n\"))]\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "template = \"\"\"You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\n",
    "Question: {question}\n",
    "Context: {context}\n",
    "Answer:\n",
    "\"\"\"\n",
    "prompt = ChatPromptTemplate.from_template(template)\n",
    "\n",
    "print(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "74982155",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e47abe3a",
   "metadata": {},
   "source": [
    "And running the cell, we get a very similar output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fe129bdd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/workspaces/langchain-weaviate/.venv/lib/python3.12/site-packages/pydantic/main.py:1024: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/\n",
      "  warnings.warn('The `dict` method is deprecated; use `model_dump` instead.', category=PydanticDeprecatedSince20)\n",
      "/workspaces/langchain-weaviate/.venv/lib/python3.12/site-packages/pydantic/main.py:1024: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/\n",
      "  warnings.warn('The `dict` method is deprecated; use `model_dump` instead.', category=PydanticDeprecatedSince20)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"The president honored Justice Stephen Breyer for his service to the country as an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. The president also mentioned nominating Circuit Court of Appeals Judge Ketanji Brown Jackson to continue Justice Breyer's legacy of excellence. The president expressed gratitude towards Justice Breyer and highlighted the importance of nominating someone to serve on the United States Supreme Court.\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "\n",
    "rag_chain = (\n",
    "    {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
    "    | prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "rag_chain.invoke(\"What did the president say about Justice Breyer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce5a2553",
   "metadata": {},
   "source": [
    "But note that since the template is upto you to construct, you can customize it to your needs. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7417ac5",
   "metadata": {},
   "source": [
    "### Wrap-up & resources\n",


"id": "0bb2affb-48ca-410b-85c0-9e1275429bcb",
   "metadata": {},
   "source": [
    "Search similar texts using similarity search function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e755cdce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='I really enjoy spending time with you', metadata={'text': 'I really enjoy spending time with you', 'id': 'text1', 'source': 'book 1', 'category': ['books', 'modern']})]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = clarifai_vector_db.similarity_search(\"I would like to see you\")\n",
    "docs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd703470-7efb-4be5-a556-eea896ca60f4",
   "metadata": {},
   "source": [
    "Further you can filter your search results by metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "140103ec-0936-454a-9f4a-7d5beefc138f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# There is lots powerful filtering you can do within an app by leveraging metadata filters.\n",
    "# This one will limit the similarity query to only the texts that have key of \"source\" matching value of \"book 1\"\n",
    "book1_similar_docs = clarifai_vector_db.similarity_search(\n",
    "    \"I would love to see you\", filter={\"source\": \"book 1\"}\n",
    ")\n",
    "\n",
    "# you can also use lists in the input's metadata and then select things that match an item in the list. This is useful for categories like below:\n",
    "book_category_similar_docs = clarifai_vector_db.similarity_search(\n",
    "    \"I would love to see you\", filter={\"category\": [\"books\"]}\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c39504e4",
   "metadata": {},
   "source": [
    "## From Documents\n",
    "Create a Clarifai vectorstore from a list of Documents. This section will upload each document with its respective metadata to a Clarifai Application. The Clarifai Application can then be used for semantic search to find relevant documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3c3999a",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = TextLoader(\"your_local_file_path.txt\")\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "40bf1305",
   "metadata": {},
   "outputs": [],
   "source": [
    "USER_ID = \"USERNAME_ID\"\n",
    "APP_ID = \"APPLICATION_ID\"\n",
    "NUMBER_OF_DOCS = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52d86f01-3462-440e-8960-3c0c17b98f09",
   "metadata": {},
   "source": [
    "Create a clarifai vector DB class and ingest all your documents into clarifai App."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e104aee",
   "metadata": {},
   "outputs": [],
   "source": [
    "clarifai_vector_db = Clarifai.from_documents(\n",
    "    user_id=USER_ID,\n",
    "    app_id=APP_ID,\n",
    "    documents=docs,\n",
    "    number_of_docs=NUMBER_OF_DOCS,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c608226",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = clarifai_vector_db.similarity_search(\"Texts related to population\")\n",
    "docs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7b332ca4-416b-4ea6-99da-b6949f399d72",
   "metadata": {},
   "source": [
    "## From existing App\n",
    "Within Clarifai we have great tools for adding data to applications (essentially projects) via API or UI. Most users will already have done that before interacting with LangChain so this example will use the data in an existing app to perform searches. Check out our [API docs](https://docs.clarifai.com/api-guide/data/create-get-update-delete) and [UI docs](https://docs.clarifai.com/portal-guide/data). The Clarifai Application can then be used for semantic search to find relevant documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "807c1141-591b-436d-abaa-f2c325e66d39",
   "metadata": {},
   "outputs": [],
   "source": [
    "USER_ID = \"USERNAME_ID\"\n",
    "APP_ID = \"APPLICATION_ID\"\n",
    "NUMBER_OF_DOCS = 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "762d74ef-f7df-43d6-b121-4980c4059fc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "clarifai_vector_db = Clarifai(\n",
    "    user_id=USER_ID,\n",
    "    app_id=APP_ID,\n",
    "    number_of_docs=NUMBER_OF_DOCS,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7636b0f-68ab-4b8f-ba0f-3c27061e3631",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = clarifai_vector_db.similarity_search(\n",
    "    \"Texts related to ammuniction and president wilson\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "55ee5fc7-94c4-45d0-84ca-00defeca871e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "683953b3",
   "metadata": {},
   "source": [
    "# Intel's Visual Data Management System (VDMS)\n",
    "\n",
    ">Intel's [VDMS](https://github.com/IntelLabs/vdms) is a storage solution for efficient access of big-”visual”-data that aims to achieve cloud scale by searching for relevant visual data via visual metadata stored as a graph and enabling machine friendly enhancements to visual data for faster access. VDMS is licensed under MIT.\n",
    "\n",
    "VDMS supports:\n",
    "* K nearest neighbor search\n",
    "* Euclidean distance (L2) and inner product (IP)\n",
    "* Libraries for indexing and computing distances: TileDBDense, TileDBSparse, FaissFlat (Default), FaissIVFFlat, Flinng\n",
    "* Embeddings for text, images, and video\n",
    "* Vector and metadata searches\n",
    "\n",
    "VDMS has server and client components. To setup the server, see the [installation instructions](https://github.com/IntelLabs/vdms/blob/master/INSTALL.md) or use the [docker image](https://hub.docker.com/r/intellabs/vdms).\n",
    "\n",
    "This notebook shows how to use VDMS as a vector store using the docker image.\n",
    "\n",
    "You'll need to install `langchain-community` with `pip install -qU langchain-community` to use this integration\n",
    "\n",
    "To begin, install the Python packages for the VDMS client and Sentence Transformers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2167badd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Pip install necessary package\n",
    "%pip install --upgrade --quiet pip vdms sentence-transformers langchain-huggingface > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af2b4512",
   "metadata": {},
   "source": [
    "## Start VDMS Server\n",
    "Here we start the VDMS server with port 55555."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4b1537c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b26917ffac236673ef1d035ab9c91fe999e29c9eb24aa6c7103d7baa6bf2f72d\n"
     ]
    }
   ],
   "source": [
    "!docker run --rm -d -p 55555:55555 --name vdms_vs_test_nb intellabs/vdms:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5ffbf8",
   "metadata": {},
   "source": [
    "## Basic Example (using the Docker Container)\n",
    "\n",
    "In this basic example, we demonstrate adding documents into VDMS and using it as a vector database.\n",
    "\n",
    "You can run the VDMS Server in a Docker container separately to use with LangChain which connects to the server via the VDMS Python Client. \n",
    "\n",
    "VDMS has the ability to handle multiple collections of documents, but the LangChain interface expects one, so we need to specify the name of the collection . The default collection name used by LangChain is \"langchain\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5201ba0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from langchain_community.document_loaders.text import TextLoader\n",
    "from langchain_community.vectorstores import VDMS\n",
    "from langchain_community.vectorstores.vdms import VDMS_Client\n",
    "from langchain_huggingface import HuggingFaceEmbeddings\n",
    "from langchain_text_splitters.character import CharacterTextSplitter\n",
    "\n",
    "time.sleep(2)\n",
    "DELIMITER = \"-\" * 50\n",
    "\n",
    "# Connect to VDMS Vector Store\n",
    "vdms_client = VDMS_Client(host=\"localhost\", port=55555)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "935069bc",
   "metadata": {},
   "source": [
    "Here are some helper functions for printing results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e78814eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_document_details(doc):\n",
    "    print(f\"Content:\\n\\t{doc.page_content}\\n\")\n",
    "    print(\"Metadata:\")\n",
    "    for key, value in doc.metadata.items():\n",
    "        if value != \"Missing property\":\n",
    "            print(f\"\\t{key}:\\t{value}\")\n",
    "\n",
    "\n",
    "def print_results(similarity_results, score=True):\n",
    "    print(f\"{DELIMITER}\\n\")\n",
    "    if score:\n",
    "        for doc, score in similarity_results:\n",
    "            print(f\"Score:\\t{score}\\n\")\n",
    "            print_document_details(doc)\n",
    "            print(f\"{DELIMITER}\\n\")\n",
    "    else:\n",
    "        for doc in similarity_results:\n",
    "            print_document_details(doc)\n",
    "            print(f\"{DELIMITER}\\n\")\n",
    "\n",
    "\n",
    "def print_response(list_of_entities):\n",
    "    for ent in list_of_entities:\n",
    "        for key, value in ent.items():\n",
    "            if value != \"Missing property\":\n",
    "                print(f\"\\n{key}:\\n\\t{value}\")\n",
    "        print(f\"{DELIMITER}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88229867",
   "metadata": {},
   "source": [
    "### Load Document and Obtain Embedding Function\n",
    "Here we load the most recent State of the Union Address and split the document into chunks. \n",
    "\n",
    "LangChain vector stores use a string/keyword `id` for bookkeeping documents. By default, `id` is a uuid but here we're defining it as an integer cast as a string. Additional metadata is also provided with the documents and the HuggingFaceEmbeddings are used for this example as the embedding function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2ebfc16c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Documents: 42\n",
      "# Embedding Dimensions: 768\n"
     ]
    }
   ],
   "source": [
    "# load the document and split it into chunks\n",
    "document_path = \"../../how_to/state_of_the_union.txt\"\n",
    "raw_documents = TextLoader(document_path).load()\n",
    "\n",
    "# split it into chunks\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(raw_documents)\n",
    "ids = []\n",
    "for doc_idx, doc in enumerate(docs):\n",
    "    ids.append(str(doc_idx + 1))\n",
    "    docs[doc_idx].metadata[\"id\"] = str(doc_idx + 1)\n",
    "    docs[doc_idx].metadata[\"page_number\"] = int(doc_idx + 1)\n",
    "    docs[doc_idx].metadata[\"president_included\"] = (\n",
    "        \"president\" in doc.page_content.lower()\n",
    "    )\n",
    "print(f\"# Documents: {len(docs)}\")\n",
    "\n",
    "\n",
    "# create the open-source embedding function\n",
    "model_name = \"sentence-transformers/all-mpnet-base-v2\"\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Azure AI Search\n",
    "\n",
    "[Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-azure-search) (formerly known as `Azure Search` and `Azure Cognitive Search`) is a cloud search service that gives developers infrastructure, APIs, and tools for information retrieval of vector, keyword, and hybrid queries at scale.\n",
    "\n",
    "You'll need to install `langchain-community` with `pip install -qU langchain-community` to use this integration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Azure AI Search SDK\n",
    "\n",
    "Use azure-search-documents package version 11.4.0 or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  azure-search-documents\n",
    "%pip install --upgrade --quiet  azure-identity"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import required libraries\n",
    "\n",
    "`OpenAIEmbeddings` is assumed, but if you're using Azure OpenAI, import `AzureOpenAIEmbeddings` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from langchain_community.vectorstores.azuresearch import AzureSearch\n",
    "from langchain_openai import AzureOpenAIEmbeddings, OpenAIEmbeddings"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure OpenAI settings\n",
    "Set variables for your OpenAI provider. You need either an [OpenAI account](https://platform.openai.com/docs/quickstart?context=python) or an [Azure OpenAI account](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource) to generate the embeddings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 1: use an OpenAI account\n",
    "openai_api_key: str = \"PLACEHOLDER FOR YOUR API KEY\"\n",
    "openai_api_version: str = \"2023-05-15\"\n",
    "model: str = \"text-embedding-ada-002\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 2: use an Azure OpenAI account with a deployment of an embedding model\n",
    "azure_endpoint: str = \"PLACEHOLDER FOR YOUR AZURE OPENAI ENDPOINT\"\n",
    "azure_openai_api_key: str = \"PLACEHOLDER FOR YOUR AZURE OPENAI KEY\"\n",
    "azure_openai_api_version: str = \"2023-05-15\"\n",
    "azure_deployment: str = \"text-embedding-ada-002\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure vector store settings\n",
    "\n",
    "You need an [Azure subscription](https://azure.microsoft.com/en-us/free/search) and [Azure AI Search service](https://learn.microsoft.com/azure/search/search-create-service-portal) to use this vector store integration. No-cost versions are available for small and limited workloads.\n",
    " \n",
    "Set variables for your Azure AI Search URL and admin API key. You can get these variables from the [Azure portal](https://portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.Search%2FsearchServices)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_store_address: str = \"YOUR_AZURE_SEARCH_ENDPOINT\"\n",
    "vector_store_password: str = \"YOUR_AZURE_SEARCH_ADMIN_KEY\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create embeddings and vector store instances\n",
    " \n",
    "Create instances of the OpenAIEmbeddings and AzureSearch classes. When you complete this step, you should have an empty search index on your Azure AI Search resource. The integration module provides a default schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 1: Use OpenAIEmbeddings with OpenAI account\n",
    "embeddings: OpenAIEmbeddings = OpenAIEmbeddings(\n",
    "    openai_api_key=openai_api_key, openai_api_version=openai_api_version, model=model\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 2: Use AzureOpenAIEmbeddings with an Azure account\n",
    "embeddings: AzureOpenAIEmbeddings = AzureOpenAIEmbeddings(\n",
    "    azure_deployment=azure_deployment,\n",
    "    openai_api_version=azure_openai_api_version,\n",
    "    azure_endpoint=azure_endpoint,\n",
    "    api_key=azure_openai_api_key,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create vector store instance\n",
    " \n",
    "Create instance of the AzureSearch class using the embeddings from above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "index_name: str = \"langchain-vector-demo\"\n",
    "vector_store: AzureSearch = AzureSearch(\n",
    "    azure_search_endpoint=vector_store_address,\n",
    "    azure_search_key=vector_store_password,\n",
    "    index_name=index_name,\n",
    "    embedding_function=embeddings.embed_query,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify additional properties for the Azure client such as the following https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/README.md#configurations\n",
    "vector_store: AzureSearch = AzureSearch(\n",
    "    azure_search_endpoint=vector_store_address,\n",
    "    azure_search_key=vector_store_password,\n",
    "    index_name=index_name,\n",
    "    embedding_function=embeddings.embed_query,\n",
    "    # Configure max retries for the Azure client\n",
    "    additional_search_client_options={\"retry_total\": 4},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert text and embeddings into vector store\n",
    " \n",
    "This step loads, chunks, and vectorizes the sample document, and then indexes the content into a search index on Azure AI Search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['M2U1OGM4YzAtYjMxYS00Nzk5LTlhNDgtZTc3MGVkNTg1Mjc0',\n",
       " 'N2I2MGNiZDEtNDdmZS00YWNiLWJhYTYtYWEzMmFiYzU1ZjZm',\n",
       " 'YWFmNDViNTQtZTc4MS00MTdjLTkzZjQtYTJkNmY1MDU4Yzll',\n",
       " 'MjgwY2ExZDctYTUxYi00NjE4LTkxMjctZDA1NDQ1MzU4NmY1',\n",
       " 'NGE4NzhkNTAtZWYxOC00ZmI5LTg0MTItZDQ1NzMxMWVmMTIz',\n",
       " 'MTYwMWU3YjAtZDIzOC00NTYwLTgwMmEtNDI1NzA2MWVhMDYz',\n",


" 'NGM5N2NlZjgtMTc5Ny00OGEzLWI5YTgtNDFiZWE2MjBlMzA0',\n",
       " 'OWQ4M2MyMTYtMmRkNi00ZDUxLWI0MDktOGE2NjMxNDFhYzFm',\n",
       " 'YWZmZGJkOTAtOGM3My00MmNiLTg5OWUtZGMwMDQwYTk1N2Vj',\n",
       " 'YTc3MTI2OTktYmVkMi00ZGU4LTgyNmUtNTY1YzZjMDg2YWI3',\n",
       " 'MTQwMmVlYjEtNDI0MS00N2E0LWEyN2ItZjhhYWU0YjllMjRk',\n",
       " 'NjJjYWY4ZjctMzgyNi00Y2I5LTkwY2UtZjRkMjJhNDQxYTFk',\n",
       " 'M2ZiM2NiYTMtM2ZiMS00YWJkLWE3ZmQtNDZiODcyOTMyYWYx',\n",
       " 'MzNmZTNkMWYtMjNmYS00Y2NmLTg3ZjQtYTZjOWM1YmJhZTRk',\n",
       " 'ZDY3MDc1NzYtY2YzZS00ZjExLWEyMjAtODhiYTRmNDUzMTBi',\n",
       " 'ZGIyYzA4NzUtZGM2Ni00MDUwLWEzZjYtNTg3MDYyOWQ5MWQy',\n",
       " 'NTA0MjBhMzYtOTYzMi00MDQ2LWExYWQtMzNiN2I4ODM4ZGZl',\n",
       " 'OTdjYzU2NGUtNWZjNC00N2ZmLWExMjQtNjhkYmZkODg4MTY3',\n",
       " 'OThhMWZmMjgtM2EzYS00OWZkLTk1NGEtZTdkNmRjNWYxYmVh',\n",
       " 'ZGVjMTQ0NzctNDVmZC00ZWY4LTg4N2EtMDQ1NWYxNWM5NDVh',\n",
       " 'MjRlYzE4YzItZTMxNy00OGY3LThmM2YtMjM0YmRhYTVmOGY3',\n",
       " 'MWU0NDA3ZDQtZDE4MS00OWMyLTlmMzktZjdkYzZhZmUwYWM3',\n",
       " 'ZGM2ZDhhY2MtM2NkNi00MzZhLWJmNTEtMmYzNjEwMzE3NmZl',\n",
       " 'YjBmMjkyZTItYTNlZC00MmY2LThiMzYtMmUxY2MyNDlhNGUw',\n",
       " 'OThmYTQ0YzEtNjk0MC00NWIyLWE1ZDQtNTI2MTZjN2NlODcw',\n",
       " 'NDdlOGU1ZGQtZTVkMi00M2MyLWExN2YtOTc2ODk3OWJmNmQw',\n",
       " 'MDVmZGNkYTUtNWI2OS00YjllLTk0YTItZDRmNWQxMWU3OTVj',\n",
       " 'YWFlNTVmNjMtMDZlNy00NmE5LWI0ODUtZTI3ZTFmZWRmNzU0',\n",
       " 'MmIzOTkxODQtODYxMi00YWM2LWFjY2YtNjRmMmEyM2JlNzMw',\n",
       " 'ZmI1NDhhNWItZWY0ZS00NTNhLWEyNDEtMTE2OWYyMjc4YTU2',\n",
       " 'YTllYTc5OTgtMzJiNC00ZjZjLWJiMzUtNWVhYzFjYzgxMjU2',\n",
       " 'ODZlZWUyOTctOGY4OS00ZjA3LWIyYTUtNDVlNDUyN2E4ZDFk',\n",
       " 'Y2M0MWRlM2YtZDU4Ny00MjZkLWE5NzgtZmRkMTNhZDg2YjEy',\n",
       " 'MDNjZWQ2ODEtMWZiMy00OTZjLTk3MzAtZjE4YjIzNWVhNTE1',\n",
       " 'OTE1NDY0NzMtODNkZS00MTk4LTk4NWQtZGVmYjQ2YjFlY2Q0',\n",
       " 'ZTgwYWQwMjEtN2ZlOS00NDk2LWIxNzUtNjk2ODE3N2U0Yzlj',\n",
       " 'ZDkxOTgzMGUtZGExMC00Yzg0LWJjMGItOWQ2ZmUwNWUwOGJj',\n",
       " 'ZGViMGI2NDEtZDdlNC00YjhiLTk0MDUtYjEyOTVlMGU1Y2I2',\n",
       " 'ODliZTYzZTctZjdlZS00YjBjLWFiZmYtMDJmNjQ0YjU3ZDcy',\n",
       " 'MDFjZGI1NzUtOTc0Ni00NWNmLThhYzYtYzRlZThkZjMwM2Vl',\n",
       " 'ZjY2ZmRiN2EtZWVhNS00ODViLTk4YjYtYjQ2Zjc4MDdkYjhk',\n",
       " 'ZTQ3NDMwODEtMTQwMy00NDFkLWJhZDQtM2UxN2RkOTU1MTdl']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "loader = TextLoader(\"../../how_to/state_of_the_union.txt\", encoding=\"utf-8\")\n",
    "\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(documents)\n",
    "\n",
    "vector_store.add_documents(documents=docs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform a vector similarity search\n",
    " \n",
    "Execute a pure vector similarity search using the similarity_search() method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
      "\n",
      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
      "\n",
      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
      "\n",
      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
     ]
    }


"This section shows you how to replace the default schema with a custom schema.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a new index with custom filterable fields \n",
    "\n",
    "This schema shows field definitions. It's the default schema, plus several new fields attributed as filterable. Because it's using the default vector configuration, you won't see vector configuration or vector profile overrides here. The name of the default vector profile is \"myHnswProfile\" and it's using a vector configuration of Hierarchical Navigable Small World (HNSW) for indexing and queries against the content_vector field.\n",
    "\n",
    "There's no data for this schema in this step. When you execute the cell, you should get an empty index on Azure AI Search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.search.documents.indexes.models import (\n",
    "    ScoringProfile,\n",
    "    SearchableField,\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    SimpleField,\n",
    "    TextWeights,\n",
    ")\n",
    "\n",
    "#  Replace OpenAIEmbeddings with AzureOpenAIEmbeddings if Azure OpenAI is your provider.\n",
    "embeddings: OpenAIEmbeddings = OpenAIEmbeddings(\n",
    "    openai_api_key=openai_api_key, openai_api_version=openai_api_version, model=model\n",
    ")\n",
    "embedding_function = embeddings.embed_query\n",
    "\n",
    "fields = [\n",
    "    SimpleField(\n",
    "        name=\"id\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        key=True,\n",
    "        filterable=True,\n",
    "    ),\n",
    "    SearchableField(\n",
    "        name=\"content\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        searchable=True,\n",
    "    ),\n",
    "    SearchField(\n",
    "        name=\"content_vector\",\n",
    "        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n",
    "        searchable=True,\n",
    "        vector_search_dimensions=len(embedding_function(\"Text\")),\n",
    "        vector_search_profile_name=\"myHnswProfile\",\n",
    "    ),\n",
    "    SearchableField(\n",
    "        name=\"metadata\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        searchable=True,\n",
    "    ),\n",
    "    # Additional field to store the title\n",
    "    SearchableField(\n",
    "        name=\"title\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        searchable=True,\n",
    "    ),\n",
    "    # Additional field for filtering on document source\n",
    "    SimpleField(\n",
    "        name=\"source\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        filterable=True,\n",
    "    ),\n",
    "]\n",
    "\n",
    "index_name: str = \"langchain-vector-demo-custom\"\n",
    "\n",
    "vector_store: AzureSearch = AzureSearch(\n",
    "    azure_search_endpoint=vector_store_address,\n",
    "    azure_search_key=vector_store_password,\n",
    "    index_name=index_name,\n",
    "    embedding_function=embedding_function,\n",
    "    fields=fields,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add data and perform a query that includes a filter\n",
    "\n",
    "This example adds data to the vector store based on the custom schema. It loads text into the title and source fields. The source field is filterable. The sample query in this section filters the results based on content in the source field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ZjhmMTg0NTEtMjgwNC00N2M0LWFiZGEtMDllMGU1Mzk1NWRm',\n",
       " 'MzQwYWUwZDEtNDJkZC00MzgzLWIwMzItYzMwOGZkYTRiZGRi',\n",
       " 'ZjFmOWVlYTQtODRiMC00YTY3LTk2YjUtMzY1NDBjNjY5ZmQ2']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data in the metadata dictionary with a corresponding field in the index will be added to the index.\n",
    "# In this example, the metadata dictionary contains a title, a source, and a random field.\n",
    "# The title and the source are added to the index as separate fields, but the random value is ignored because it's not defined in the schema.\n",
    "# The random field is only stored in the metadata field.\n",
    "vector_store.add_texts(\n",
    "    [\"Test 1\", \"Test 2\", \"Test 3\"],\n",
    "    [\n",
    "        {\"title\": \"Title 1\", \"source\": \"A\", \"random\": \"10290\"},\n",
    "        {\"title\": \"Title 2\", \"source\": \"A\", \"random\": \"48392\"},\n",
    "        {\"title\": \"Title 3\", \"source\": \"B\", \"random\": \"32893\"},\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Test 3', metadata={'title': 'Title 3', 'source': 'B', 'random': '32893'}),\n",
       " Document(page_content='Test 1', metadata={'title': 'Title 1', 'source': 'A', 'random': '10290'}),\n",
       " Document(page_content='Test 2', metadata={'title': 'Title 2', 'source': 'A', 'random': '48392'})]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = vector_store.similarity_search(query=\"Test 3 source1\", k=3, search_type=\"hybrid\")\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Test 1', metadata={'title': 'Title 1', 'source': 'A', 'random': '10290'}),\n",
       " Document(page_content='Test 2', metadata={'title': 'Title 2', 'source': 'A', 'random': '48392'})]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = vector_store.similarity_search(\n",
    "    query=\"Test 3 source1\", k=3, search_type=\"hybrid\", filters=\"source eq 'A'\"\n",
    ")\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a new index with a scoring profile\n",
    "\n",
    "Here's another custom schema that includes a scoring profile definition. A scoring profile is used for relevance tuning of nonvector content, which is helpful in hybrid search scenarios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.search.documents.indexes.models import (\n",
    "    FreshnessScoringFunction,\n",
    "    FreshnessScoringParameters,\n",
    "    ScoringProfile,\n",
    "    SearchableField,\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    SimpleField,\n",
    "    TextWeights,\n",
    ")\n",
    "\n",
    "#  Replace OpenAIEmbeddings with AzureOpenAIEmbeddings if Azure OpenAI is your provider.\n",


"embeddings: OpenAIEmbeddings = OpenAIEmbeddings(\n",
    "    openai_api_key=openai_api_key, openai_api_version=openai_api_version, model=model\n",
    ")\n",
    "embedding_function = embeddings.embed_query\n",
    "\n",
    "fields = [\n",
    "    SimpleField(\n",
    "        name=\"id\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        key=True,\n",
    "        filterable=True,\n",
    "    ),\n",
    "    SearchableField(\n",
    "        name=\"content\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        searchable=True,\n",
    "    ),\n",
    "    SearchField(\n",
    "        name=\"content_vector\",\n",
    "        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n",
    "        searchable=True,\n",
    "        vector_search_dimensions=len(embedding_function(\"Text\")),\n",
    "        vector_search_profile_name=\"myHnswProfile\",\n",
    "    ),\n",
    "    SearchableField(\n",
    "        name=\"metadata\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        searchable=True,\n",
    "    ),\n",
    "    # Additional field to store the title\n",
    "    SearchableField(\n",
    "        name=\"title\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        searchable=True,\n",
    "    ),\n",
    "    # Additional field for filtering on document source\n",
    "    SimpleField(\n",
    "        name=\"source\",\n",
    "        type=SearchFieldDataType.String,\n",
    "        filterable=True,\n",
    "    ),\n",
    "    # Additional data field for last doc update\n",
    "    SimpleField(\n",
    "        name=\"last_update\",\n",
    "        type=SearchFieldDataType.DateTimeOffset,\n",
    "        searchable=True,\n",
    "        filterable=True,\n",
    "    ),\n",
    "]\n",
    "# Adding a custom scoring profile with a freshness function\n",
    "sc_name = \"scoring_profile\"\n",
    "sc = ScoringProfile(\n",
    "    name=sc_name,\n",
    "    text_weights=TextWeights(weights={\"title\": 5}),\n",
    "    function_aggregation=\"sum\",\n",
    "    functions=[\n",
    "        FreshnessScoringFunction(\n",
    "            field_name=\"last_update\",\n",
    "            boost=100,\n",
    "            parameters=FreshnessScoringParameters(boosting_duration=\"P2D\"),\n",
    "            interpolation=\"linear\",\n",
    "        )\n",
    "    ],\n",
    ")\n",
    "\n",
    "index_name = \"langchain-vector-demo-custom-scoring-profile\"\n",
    "\n",
    "vector_store: AzureSearch = AzureSearch(\n",
    "    azure_search_endpoint=vector_store_address,\n",
    "    azure_search_key=vector_store_password,\n",
    "    index_name=index_name,\n",
    "    embedding_function=embeddings.embed_query,\n",
    "    fields=fields,\n",
    "    scoring_profiles=[sc],\n",
    "    default_scoring_profile=sc_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['NjUwNGQ5ZDUtMGVmMy00OGM4LWIxMGYtY2Y2MDFmMTQ0MjE5',\n",
       " 'NWFjN2YwY2UtOWQ4Yi00OTNhLTg2MGEtOWE0NGViZTVjOGRh',\n",
       " 'N2Y2NWUyZjctMDBjZC00OGY4LWJlZDEtNTcxYjQ1MmI1NjYx']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Adding same data with different last_update to show Scoring Profile effect\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "today = datetime.utcnow().strftime(\"%Y-%m-%dT%H:%M:%S-00:00\")\n",
    "yesterday = (datetime.utcnow() - timedelta(days=1)).strftime(\"%Y-%m-%dT%H:%M:%S-00:00\")\n",
    "one_month_ago = (datetime.utcnow() - timedelta(days=30)).strftime(\n",
    "    \"%Y-%m-%dT%H:%M:%S-00:00\"\n",
    ")\n",
    "\n",
    "vector_store.add_texts(\n",
    "    [\"Test 1\", \"Test 1\", \"Test 1\"],\n",
    "    [\n",
    "        {\n",
    "            \"title\": \"Title 1\",\n",
    "            \"source\": \"source1\",\n",
    "            \"random\": \"10290\",\n",
    "            \"last_update\": today,\n",
    "        },\n",
    "        {\n",
    "            \"title\": \"Title 1\",\n",
    "            \"source\": \"source1\",\n",
    "            \"random\": \"48392\",\n",
    "            \"last_update\": yesterday,\n",
    "        },\n",
    "        {\n",
    "            \"title\": \"Title 1\",\n",
    "            \"source\": \"source1\",\n",
    "            \"random\": \"32893\",\n",
    "            \"last_update\": one_month_ago,\n",
    "        },\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Test 1', metadata={'title': 'Title 1', 'source': 'source1', 'random': '32893', 'last_update': '2024-01-24T22:18:51-00:00'}),\n",
       " Document(page_content='Test 1', metadata={'title': 'Title 1', 'source': 'source1', 'random': '48392', 'last_update': '2024-02-22T22:18:51-00:00'}),\n",
       " Document(page_content='Test 1', metadata={'title': 'Title 1', 'source': 'source1', 'random': '10290', 'last_update': '2024-02-23T22:18:51-00:00'})]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = vector_store.similarity_search(query=\"Test 1\", k=3, search_type=\"similarity\")\n",
    "res"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 ('.venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "645053d6307d413a1a75681b5ebb6449bb2babba4bcb0bf65a1ddc3dbefb108a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upstash Vector\n",
    "\n",
    "> [Upstash Vector](https://upstash.com/docs/vector/overall/whatisvector) is a serverless vector database designed for working with vector embeddings.\n",
    ">\n",
    "> The vector langchain integration is a wrapper around the [upstash-vector](https://github.com/upstash/vector-py) package.\n",
    ">\n",
    "> The python package uses the [vector rest api](https://upstash.com/docs/vector/api/get-started) behind the scenes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "Create a free vector database from [upstash console](https://console.upstash.com/vector) with the desired dimensions and distance metric.\n",
    "\n",
    "You can then create an `UpstashVectorStore` instance by:\n",
    "\n",
    "- Providing the environment variables `UPSTASH_VECTOR_URL` and `UPSTASH_VECTOR_TOKEN`\n",
    "\n",
    "- Giving them as parameters to the constructor\n",
    "\n",
    "- Passing an Upstash Vector `Index` instance to the constructor\n",
    "\n",
    "Also, an `Embeddings` instance is required to turn given texts into embeddings. Here we use `OpenAIEmbeddings` as an example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install langchain-openai langchain langchain-community upstash-vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from langchain_community.vectorstores.upstash import UpstashVectorStore\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"<YOUR_OPENAI_KEY>\"\n",
    "os.environ[\"UPSTASH_VECTOR_REST_URL\"] = \"<YOUR_UPSTASH_VECTOR_URL>\"\n",
    "os.environ[\"UPSTASH_VECTOR_REST_TOKEN\"] = \"<YOUR_UPSTASH_VECTOR_TOKEN>\"\n",
    "\n",
    "# Create an embeddings instance\n",
    "embeddings = OpenAIEmbeddings()\n",
    "\n",
    "# Create a vector store instance\n",
    "store = UpstashVectorStore(embedding=embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative way of creating `UpstashVectorStore` is to [create an Upstash Vector index by selecting a model](https://upstash.com/docs/vector/features/embeddingmodels#using-a-model) and passing `embedding=True`. In this configuration, documents or queries will be sent to Upstash as text and embedded there.\n",
    "\n",
    "```python\n",
    "store = UpstashVectorStore(embedding=True)\n",
    "```\n",
    "\n",
    "If you are interested in trying out this approach, you can update the initialization of `store` like above and run the rest of the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load documents\n",
    "\n",
    "Load an example text file and split it into chunks which can be turned into vector embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': '../../how_to/state_of_the_union.txt'}, page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\nWith a duty to one another to the American people to the Constitution. \\n\\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \\n\\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \\n\\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \\n\\nHe met the Ukrainian people. \\n\\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.'),\n",
       " Document(metadata={'source': '../../how_to/state_of_the_union.txt'}, page_content='Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. \\n\\nIn this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. \\n\\nLet each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. \\n\\nPlease rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. \\n\\nThroughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos.   \\n\\nThey keep moving.   \\n\\nAnd the costs and the threats to America and the world keep rising.   \\n\\nThat’s why the NATO Alliance was created to secure peace and stability in Europe after World War 2. \\n\\nThe United States is a member along with 29 other nations. \\n\\nIt matters. American diplomacy matters. American resolve matters.'),\n",
       " Document(metadata={'source': '../../how_to/state_of_the_union.txt'}, page_content='Putin’s latest attack on Ukraine was premeditated and unprovoked. \\n\\nHe rejected repeated efforts at diplomacy. \\n\\nHe thought the West and NATO wouldn’t respond. And he thought he could divide us at home. Putin was wrong. We were ready.  Here is what we did.   \\n\\nWe prepared extensively and carefully. \\n\\nWe spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin. \\n\\nI spent countless hours unifying our European allies. We shared with the world in advance what we knew Putin was planning and precisely how he would try to falsely justify his aggression.  \\n\\nWe countered Russia’s lies with truth.   \\n\\nAnd now that he has acted the free world is holding him accountable. \\n\\nAlong with twenty-seven members of the European Union including France, Germany, Italy, as well as countries like the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "loader = TextLoader(\"../../how_to/state_of_the_union.txt\")\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(documents)\n",
    "\n",
    "docs[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inserting documents\n",
    "\n",
    "The vectorstore embeds text chunks using the embedding object and batch inserts them into the database. This returns an id array of the inserted vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['247aa3ae-9be9-43e2-98e4-48f94f920749',\n",
       " 'c4dfc886-0a2d-497c-b2b7-d923a5cb3832',\n",
       " '0350761d-ca68-414e-b8db-7eca78cb0d18',\n",
       " '902fe5eb-8543-486a-bd5f-79858a7a8af1',\n",
       " '28875612-c672-4de4-b40a-3b658c72036a']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inserted_vectors = store.add_documents(docs)\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Activeloop Deep Lake\n",
    "\n",
    ">[Activeloop Deep Lake](https://docs.activeloop.ai/) as a Multi-Modal Vector Store that stores embeddings and their metadata including text, Jsons, images, audio, video, and more. It saves the data locally, in your cloud, or on Activeloop storage. It performs hybrid search including embeddings and their attributes.\n",
    "\n",
    "This notebook showcases basic functionality related to `Activeloop Deep Lake`. While `Deep Lake` can store embeddings, it is capable of storing any type of data. It is a serverless data lake with version control, query engine and streaming dataloaders to deep learning frameworks.  \n",
    "\n",
    "For more information, please see the Deep Lake [documentation](https://docs.activeloop.ai) or [api reference](https://docs.deeplake.ai)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain-openai langchain-community 'deeplake[enterprise]' tiktoken"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example provided by Activeloop\n",
    "\n",
    "[Integration with LangChain](https://docs.activeloop.ai/tutorials/vector-store/deep-lake-vector-store-in-langchain).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Lake locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.vectorstores import DeepLake\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
    "activeloop_token = getpass.getpass(\"activeloop token:\")\n",
    "embeddings = OpenAIEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "\n",
    "loader = TextLoader(\"../../how_to/state_of_the_union.txt\")\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(documents)\n",
    "\n",
    "embeddings = OpenAIEmbeddings()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a local dataset\n",
    "\n",
    "Create a dataset locally at `./deeplake/`, then run similarity search. The Deeplake+LangChain integration uses Deep Lake datasets under the hood, so `dataset` and `vector store` are used interchangeably. To create a dataset in your own cloud, or in the Deep Lake storage, [adjust the path accordingly](https://docs.activeloop.ai/storage-and-credentials/storage-options)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = DeepLake(dataset_path=\"./my_deeplake/\", embedding=embeddings, overwrite=True)\n",
    "db.add_documents(docs)\n",
    "# or shorter\n",
    "# db = DeepLake.from_documents(docs, dataset_path=\"./my_deeplake/\", embedding=embeddings, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset(path='./my_deeplake/', tensors=['embedding', 'id', 'metadata', 'text'])\n",
      "\n",
      "  tensor      htype      shape      dtype  compression\n",
      "  -------    -------    -------    -------  ------- \n",
      " embedding  embedding  (42, 1536)  float32   None   \n",
      "    id        text      (42, 1)      str     None   \n",
      " metadata     json      (42, 1)      str     None   \n",
      "   text       text      (42, 1)      str     None   \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
    "docs = db.similarity_search(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To disable dataset summary printings all the time, you can specify verbose=False during VectorStore initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
      "\n",
      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
      "\n",
      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
      "\n",
      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
     ]
    }
   ],
   "source": [
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later, you can reload the dataset without recomputing embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep Lake Dataset in ./my_deeplake/ already exists, loading from the storage\n"
     ]
    }
   ],
   "source": [
    "db = DeepLake(dataset_path=\"./my_deeplake/\", embedding=embeddings, read_only=True)\n",
    "docs = db.similarity_search(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Lake, for now, is single writer and multiple reader. Setting `read_only=True` helps to avoid acquiring the writer lock."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieval Question/Answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/langchain_activeloop/langchain/libs/langchain/langchain/llms/openai.py:786: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain_openai import ChatOpenAI`\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain.chains import RetrievalQA\n",


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fb0243ae",
   "metadata": {},
   "source": [
    "# Azure Cosmos DB No SQL\n",
    "\n",
    "This notebook shows you how to leverage this integrated [vector database](https://learn.microsoft.com/en-us/azure/cosmos-db/vector-database) to store documents in collections, create indicies and perform vector search queries using approximate nearest neighbor algorithms such as COS (cosine distance), L2 (Euclidean distance), and IP (inner product) to locate documents close to the query vectors. \n",
    "    \n",
    "Azure Cosmos DB is the database that powers OpenAI's ChatGPT service. It offers single-digit millisecond response times, automatic and instant scalability, along with guaranteed speed at any scale. \n",
    "\n",
    "[Azure Cosmos DB for NoSQL](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/vector-search) now offers vector indexing and search in preview. This feature is designed to handle high-dimensional vectors, enabling efficient and accurate vector search at any scale. You can now store vectors directly in the documents alongside your data. This means that each document in your database can contain not only traditional schema-free data, but also high-dimensional vectors as other properties of the documents. This colocation of data and vectors allows for efficient indexing and searching, as the vectors are stored in the same logical unit as the data they represent. This simplifies data management, AI application architectures, and the efficiency of vector-based operations.\n",
    "\n",
    "[Sign Up](https://azure.microsoft.com/en-us/free/) for lifetime free access to get started today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ad3c1e88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade --quiet azure-cosmos langchain-openai langchain-community"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c507b0e8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-25T01:36:53.595385Z",
     "start_time": "2024-05-25T01:36:53.571737Z"
    }
   },
   "outputs": [],
   "source": [
    "OPENAI_API_KEY = \"YOUR_KEY\"\n",
    "OPENAI_API_TYPE = \"azure\"\n",
    "OPENAI_API_VERSION = \"2023-05-15\"\n",
    "OPENAI_API_BASE = \"YOUR_ENDPOINT\"\n",
    "OPENAI_EMBEDDINGS_MODEL_NAME = \"text-embedding-ada-002\"\n",
    "OPENAI_EMBEDDINGS_MODEL_DEPLOYMENT = \"text-embedding-ada-002\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa7101f64740fb76",
   "metadata": {},
   "source": [
    "## Insert Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "8205cd27",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-25T01:43:02.731634Z",
     "start_time": "2024-05-25T01:43:00.383956Z"
    }
   },
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "# Load the PDF\n",
    "loader = PyPDFLoader(\"https://arxiv.org/pdf/2303.08774.pdf\")\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "8d33cceb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-25T01:43:02.787966Z",
     "start_time": "2024-05-25T01:43:02.763502Z"
    }
   },
   "outputs": [],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)\n",
    "docs = text_splitter.split_documents(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "6a80f1c2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-25T01:43:04.582560Z",
     "start_time": "2024-05-25T01:43:04.578948Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='GPT-4 Technical Report\\nOpenAI∗\\nAbstract\\nWe report the development of GPT-4, a large-scale, multimodal model which can\\naccept image and text inputs and produce text outputs. While less capable than\\nhumans in many real-world scenarios, GPT-4 exhibits human-level performance\\non various professional and academic benchmarks, including passing a simulated\\nbar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-\\nbased model pre-trained to predict the next token in a document. The post-training\\nalignment process results in improved performance on measures of factuality and\\nadherence to desired behavior. A core component of this project was developing\\ninfrastructure and optimization methods that behave predictably across a wide\\nrange of scales. This allowed us to accurately predict some aspects of GPT-4’s\\nperformance based on models trained with no more than 1/1,000th the compute of\\nGPT-4.\\n1 Introduction' metadata={'source': 'https://arxiv.org/pdf/2303.08774.pdf', 'page': 0}\n"
     ]
    }
   ],
   "source": [
    "print(docs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd1f13e237e91052",
   "metadata": {},
   "source": [
    "## Creating AzureCosmosDB NoSQL Vector Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "04c72ccc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-25T01:43:13.279497Z",
     "start_time": "2024-05-25T01:43:13.275379Z"
    }
   },
   "outputs": [],
   "source": [
    "indexing_policy = {\n",
    "    \"indexingMode\": \"consistent\",\n",
    "    \"includedPaths\": [{\"path\": \"/*\"}],\n",
    "    \"excludedPaths\": [{\"path\": '/\"_etag\"/?'}],\n",
    "    \"vectorIndexes\": [{\"path\": \"/embedding\", \"type\": \"quantizedFlat\"}],\n",
    "}\n",
    "\n",
    "vector_embedding_policy = {\n",
    "    \"vectorEmbeddings\": [\n",
    "        {\n",
    "            \"path\": \"/embedding\",\n",
    "            \"dataType\": \"float32\",\n",
    "            \"distanceFunction\": \"cosine\",\n",
    "            \"dimensions\": 1536,\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "4ebad8ef01a6c04f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-05-25T01:48:42.981276Z",
     "start_time": "2024-05-25T01:44:55.468667Z"
    }
   },
   "outputs": [],
   "source": [
    "from azure.cosmos import CosmosClient, PartitionKey\n",
    "from langchain_community.vectorstores.azure_cosmos_db_no_sql import (\n",
    "    AzureCosmosDBNoSqlVectorSearch,\n",
    ")\n",
    "from langchain_openai import AzureOpenAIEmbeddings\n",
    "\n",
    "HOST = \"AZURE_COSMOS_DB_ENDPOINT\"\n",
    "KEY = \"AZURE_COSMOS_DB_KEY\"\n",
    "\n",
    "cosmos_client = CosmosClient(HOST, KEY)\n",


"source": [
    "## Basic Vectorstore Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = HanaDB(\n",
    "    connection=connection, embedding=embeddings, table_name=\"LANGCHAIN_DEMO_BASIC\"\n",
    ")\n",
    "\n",
    "# Delete already existing documents from the table\n",
    "db.delete(filter={})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add simple text documents to the existing table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = [Document(page_content=\"Some text\"), Document(page_content=\"Other docs\")]\n",
    "db.add_documents(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add documents with metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = [\n",
    "    Document(\n",
    "        page_content=\"foo\",\n",
    "        metadata={\"start\": 100, \"end\": 150, \"doc_name\": \"foo.txt\", \"quality\": \"bad\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"bar\",\n",
    "        metadata={\"start\": 200, \"end\": 250, \"doc_name\": \"bar.txt\", \"quality\": \"good\"},\n",
    "    ),\n",
    "]\n",
    "db.add_documents(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Query documents with specific metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = db.similarity_search(\"foobar\", k=2, filter={\"quality\": \"bad\"})\n",
    "# With filtering on \"quality\"==\"bad\", only one document should be returned\n",
    "for doc in docs:\n",
    "    print(\"-\" * 80)\n",
    "    print(doc.page_content)\n",
    "    print(doc.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete documents with specific metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "db.delete(filter={\"quality\": \"bad\"})\n",
    "\n",
    "# Now the similarity search with the same filter will return no results\n",
    "docs = db.similarity_search(\"foobar\", k=2, filter={\"quality\": \"bad\"})\n",
    "print(len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced filtering\n",
    "In addition to the basic value-based filtering capabilities, it is possible to use more advanced filtering.\n",
    "The table below shows the available filter operators.\n",
    "\n",
    "| Operator | Semantic                 |\n",
    "|----------|-------------------------|\n",
    "| `$eq`    | Equality (==)           |\n",
    "| `$ne`    | Inequality (!=)         |\n",
    "| `$lt`    | Less than (<)           |\n",
    "| `$lte`   | Less than or equal (<=) |\n",
    "| `$gt`    | Greater than (>)        |\n",
    "| `$gte`   | Greater than or equal (>=) |\n",
    "| `$in`    | Contained in a set of given values  (in)    |\n",
    "| `$nin`   | Not contained in a set of given values  (not in)  |\n",
    "| `$between` | Between the range of two boundary values |\n",
    "| `$like`  | Text equality based on the \"LIKE\" semantics in SQL (using \"%\" as wildcard)  |\n",
    "| `$and`   | Logical \"and\", supporting 2 or more operands |\n",
    "| `$or`    | Logical \"or\", supporting 2 or more operands |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare some test documents\n",
    "docs = [\n",
    "    Document(\n",
    "        page_content=\"First\",\n",
    "        metadata={\"name\": \"adam\", \"is_active\": True, \"id\": 1, \"height\": 10.0},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Second\",\n",
    "        metadata={\"name\": \"bob\", \"is_active\": False, \"id\": 2, \"height\": 5.7},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Third\",\n",
    "        metadata={\"name\": \"jane\", \"is_active\": True, \"id\": 3, \"height\": 2.4},\n",
    "    ),\n",
    "]\n",
    "\n",
    "db = HanaDB(\n",
    "    connection=connection,\n",
    "    embedding=embeddings,\n",
    "    table_name=\"LANGCHAIN_DEMO_ADVANCED_FILTER\",\n",
    ")\n",
    "\n",
    "# Delete already existing documents from the table\n",
    "db.delete(filter={})\n",
    "db.add_documents(docs)\n",
    "\n",
    "\n",
    "# Helper function for printing filter results\n",
    "def print_filter_result(result):\n",
    "    if len(result) == 0:\n",
    "        print(\"<empty result>\")\n",
    "    for doc in result:\n",
    "        print(doc.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filtering with `$ne`, `$gt`, `$gte`, `$lt`, `$lte`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "advanced_filter = {\"id\": {\"$ne\": 1}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"id\": {\"$gt\": 1}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"id\": {\"$gte\": 1}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"id\": {\"$lt\": 1}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"id\": {\"$lte\": 1}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filtering with `$between`, `$in`, `$nin`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "advanced_filter = {\"id\": {\"$between\": (1, 2)}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"name\": {\"$in\": [\"adam\", \"bob\"]}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",


"advanced_filter = {\"name\": {\"$nin\": [\"adam\", \"bob\"]}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text filtering with `$like`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "advanced_filter = {\"name\": {\"$like\": \"a%\"}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"name\": {\"$like\": \"%a%\"}}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combined filtering with `$and`, `$or`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "advanced_filter = {\"$or\": [{\"id\": 1}, {\"name\": \"bob\"}]}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"$and\": [{\"id\": 1}, {\"id\": 2}]}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))\n",
    "\n",
    "advanced_filter = {\"$or\": [{\"id\": 1}, {\"id\": 2}, {\"id\": 3}]}\n",
    "print(f\"Filter: {advanced_filter}\")\n",
    "print_filter_result(db.similarity_search(\"just testing\", k=5, filter=advanced_filter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a VectorStore as a retriever in chains for retrieval augmented generation (RAG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.memory import ConversationBufferMemory\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "# Access the vector DB with a new table\n",
    "db = HanaDB(\n",
    "    connection=connection,\n",
    "    embedding=embeddings,\n",
    "    table_name=\"LANGCHAIN_DEMO_RETRIEVAL_CHAIN\",\n",
    ")\n",
    "\n",
    "# Delete already existing entries from the table\n",
    "db.delete(filter={})\n",
    "\n",
    "# add the loaded document chunks from the \"State Of The Union\" file\n",
    "db.add_documents(text_chunks)\n",
    "\n",
    "# Create a retriever instance of the vector store\n",
    "retriever = db.as_retriever()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "prompt_template = \"\"\"\n",
    "You are an expert in state of the union topics. You are provided multiple context items that are related to the prompt you have to answer.\n",
    "Use the following pieces of context to answer the question at the end.\n",
    "\n",
    "'''\n",
    "{context}\n",
    "'''\n",
    "\n",
    "Question: {question}\n",
    "\"\"\"\n",
    "\n",
    "PROMPT = PromptTemplate(\n",
    "    template=prompt_template, input_variables=[\"context\", \"question\"]\n",
    ")\n",
    "chain_type_kwargs = {\"prompt\": PROMPT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the ConversationalRetrievalChain, which handles the chat history and the retrieval of similar document chunks to be added to the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import ConversationalRetrievalChain\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\")\n",
    "memory = ConversationBufferMemory(\n",
    "    memory_key=\"chat_history\", output_key=\"answer\", return_messages=True\n",
    ")\n",
    "qa_chain = ConversationalRetrievalChain.from_llm(\n",
    "    llm,\n",
    "    db.as_retriever(search_kwargs={\"k\": 5}),\n",
    "    return_source_documents=True,\n",
    "    memory=memory,\n",
    "    verbose=False,\n",
    "    combine_docs_chain_kwargs={\"prompt\": PROMPT},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ask the first question (and verify how many text chunks have been used)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"What about Mexico and Guatemala?\"\n",
    "\n",
    "result = qa_chain.invoke({\"question\": question})\n",
    "print(\"Answer from LLM:\")\n",
    "print(\"================\")\n",
    "print(result[\"answer\"])\n",
    "\n",
    "source_docs = result[\"source_documents\"]\n",
    "print(\"================\")\n",
    "print(f\"Number of used source document chunks: {len(source_docs)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the used chunks of the chain in detail. Check if the best ranked chunk contains info about \"Mexico and Guatemala\" as mentioned in the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in source_docs:\n",
    "    print(\"-\" * 80)\n",
    "    print(doc.page_content)\n",
    "    print(doc.metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ask another question on the same conversational chain. The answer should relate to the previous answer given."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"What about other countries?\"\n",
    "\n",
    "result = qa_chain.invoke({\"question\": question})\n",
    "print(\"Answer from LLM:\")\n",
    "print(\"================\")\n",
    "print(result[\"answer\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard tables vs. \"custom\" tables with vector data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As default behaviour, the table for the embeddings is created with 3 columns:\n",
    "\n",
    "- A column `VEC_TEXT`, which contains the text of the Document\n",
    "- A column `VEC_META`, which contains the metadata of the Document\n",
    "- A column `VEC_VECTOR`, which contains the embeddings-vector of the Document's text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Access the vector DB with a new table\n",
    "db = HanaDB(\n",
    "    connection=connection, embedding=embeddings, table_name=\"LANGCHAIN_DEMO_NEW_TABLE\"\n",
    ")\n",
    "\n",
    "# Delete already existing entries from the table\n",
    "db.delete(filter={})\n",
    "\n",


ScaNN (Scalable Nearest Neighbors) is a method for efficient vector similarity search at scale.

ScaNN includes search space pruning and quantization for Maximum Inner Product Search and also supports other distance functions such as Euclidean distance. The implementation is optimized for x86 processors with AVX2 support. See its [Google Research github](https://github.com/google-research/google-research/tree/master/scann) for more details.

You'll need to install `langchain-community` with `pip install -qU langchain-community` to use this integration

Install ScaNN through pip. Alternatively, you can follow instructions on the [ScaNN Website](https://github.com/google-research/google-research/tree/master/scann#building-from-source) to install from source.

Below we show how to use ScaNN in conjunction with Huggingface Embeddings.

from langchain_community.document_loaders import TextLoader

from langchain_community.vectorstores import ScaNN

from langchain_huggingface import HuggingFaceEmbeddings

from langchain_text_splitters import CharacterTextSplitter

text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)

db = ScaNN.from_documents(docs, embeddings)

query = "What did the president say about Ketanji Brown Jackson"

Next, we demonstrate using ScaNN in conjunction with Google PaLM API.

You can obtain an API key from https://developers.generativeai.google/tutorials/setup

from langchain.chains import RetrievalQA

from langchain_community.chat_models.google_palm import ChatGooglePalm

stdout

The president said that Ketanji Brown Jackson is one of our nation's top legal minds, who will continue Justice Breyer's legacy of excellence.

print(qa.run("What did the president say about Ketanji Brown Jackson?"))

The president did not mention Michael Phelps in his speech.

print(qa.run("What did the president say about Michael Phelps?"))

## Save and loading local retrieval index

restored_db = ScaNN.load_local("/tmp/db", embeddings, index_name="state_of_union")


{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google Memorystore for Redis\n",
    "\n",
    "> [Google Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) is a fully-managed service that is powered by the Redis in-memory data store to build application caches that provide sub-millisecond data access. Extend your database application to build AI-powered experiences leveraging Memorystore for Redis's Langchain integrations.\n",
    "\n",
    "This notebook goes over how to use [Memorystore for Redis](https://cloud.google.com/memorystore/docs/redis/memorystore-for-redis-overview) to store vector embeddings with the `MemorystoreVectorStore` class.\n",
    "\n",
    "Learn more about the package on [GitHub](https://github.com/googleapis/langchain-google-memorystore-redis-python/).\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googleapis/langchain-google-memorystore-redis-python/blob/main/docs/vector_store.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-reqs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before You Begin\n",
    "\n",
    "To run this notebook, you will need to do the following:\n",
    "\n",
    "* [Create a Google Cloud Project](https://developers.google.com/workspace/guides/create-project)\n",
    "* [Enable the Memorystore for Redis API](https://console.cloud.google.com/flows/enableapi?apiid=redis.googleapis.com)\n",
    "* [Create a Memorystore for Redis instance](https://cloud.google.com/memorystore/docs/redis/create-instance-console). Ensure that the version is greater than or equal to 7.2."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🦜🔗 Library Installation\n",
    "\n",
    "The integration lives in its own `langchain-google-memorystore-redis` package, so we need to install it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install -upgrade --quiet langchain-google-memorystore-redis langchain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Colab only:** Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☁ Set Your Google Cloud Project\n",
    "Set your Google Cloud project so that you can leverage Google Cloud resources within this notebook.\n",
    "\n",
    "If you don't know your project ID, try the following:\n",
    "\n",
    "* Run `gcloud config list`.\n",
    "* Run `gcloud projects list`.\n",
    "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @markdown Please fill in the value below with your Google Cloud project ID and then run the cell.\n",
    "\n",
    "PROJECT_ID = \"my-project-id\"  # @param {type:\"string\"}\n",
    "\n",
    "# Set the project id\n",
    "!gcloud config set project {PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🔐 Authentication\n",
    "Authenticate to Google Cloud as the IAM user logged into this notebook in order to access your Google Cloud Project.\n",
    "\n",
    "* If you are using Colab to run this notebook, use the cell below and continue.\n",
    "* If you are using Vertex AI Workbench, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import auth\n",
    "\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Usage"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize a Vector Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import redis\n",
    "from langchain_google_memorystore_redis import (\n",
    "    DistanceStrategy,\n",
    "    HNSWConfig,\n",
    "    RedisVectorStore,\n",
    ")\n",
    "\n",
    "# Connect to a Memorystore for Redis instance\n",
    "redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n",
    "\n",
    "# Configure HNSW index with descriptive parameters\n",
    "index_config = HNSWConfig(\n",
    "    name=\"my_vector_index\", distance_strategy=DistanceStrategy.COSINE, vector_size=128\n",
    ")\n",
    "\n",
    "# Initialize/create the vector store index\n",
    "RedisVectorStore.init_index(client=redis_client, index_config=index_config)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Documents\n",
    "\n",
    "Text needs processing and numerical representation before interacting with a vector store. This involves:\n",
    "\n",
    "* Loading Text: The TextLoader obtains text data from a file (e.g., \"state_of_the_union.txt\").\n",
    "* Text Splitting: The CharacterTextSplitter breaks the text into smaller chunks for embedding models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "loader = TextLoader(\"./state_of_the_union.txt\")\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(documents)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Documents to the Vector Store\n",
    "\n",
    "After text preparation and embedding generation, the following methods insert them into the Redis vector store."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 1: Classmethod for Direct Insertion\n",
    "\n",
    "This approach combines embedding creation and insertion into a single step using the from_documents classmethod:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.embeddings.fake import FakeEmbeddings\n",
    "\n",
    "embeddings = FakeEmbeddings(size=128)\n",
    "redis_client = redis.from_url(\"redis://127.0.0.1:6379\")\n",
    "rvs = RedisVectorStore.from_documents(\n",
    "    docs, embedding=embeddings, client=redis_client, index_name=\"my_vector_index\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 2: Instance-Based Insertion\n",