{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure chat completion models with your own data (preview)\n",
    "\n",
    "> Note: There is a newer version of the openai library available. See https://github.com/openai/openai-python/discussions/742\n",
    "\n",
    "This example shows how to use Azure OpenAI service models with your own data. The feature is currently in preview. \n",
    "\n",
    "Azure OpenAI on your data enables you to run supported chat models such as GPT-3.5-Turbo and GPT-4 on your data without needing to train or fine-tune models. Running models on your data enables you to chat on top of, and analyze your data with greater accuracy and speed. One of the key benefits of Azure OpenAI on your data is its ability to tailor the content of conversational AI. Because the model has access to, and can reference specific sources to support its responses, answers are not only based on its pretrained knowledge but also on the latest information available in the designated data source. This grounding data also helps the model avoid generating responses based on outdated or incorrect information.\n",
    "\n",
    "Azure OpenAI on your own data with Azure Cognitive Search provides a customizable, pre-built solution for knowledge retrieval, from which a conversational AI application can be built. To see alternative methods for knowledge retrieval and semantic search, check out the cookbook examples for [vector databases](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works\n",
    "\n",
    "[Azure OpenAI on your own data](https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data) connects the model with your data, giving it the ability to retrieve and utilize data in a way that enhances the model's output. Together with Azure Cognitive Search, data is retrieved from designated data sources based on the user input and provided conversation history. The data is then augmented and resubmitted as a prompt to the model, giving the model contextual information it can use to generate a response.\n",
    "\n",
    "See the [Data, privacy, and security for Azure OpenAI Service](https://learn.microsoft.com/legal/cognitive-services/openai/data-privacy?context=%2Fazure%2Fai-services%2Fopenai%2Fcontext%2Fcontext) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "To get started, we'll cover a few prequisites. \n",
    "\n",
    "To properly access the Azure OpenAI Service, we need to create the proper resources at the [Azure Portal](https://portal.azure.com) (you can check a detailed guide on how to do this in the [Microsoft Docs](https://learn.microsoft.com/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal))\n",
    "\n",
    "To use your own data with Azure OpenAI models, you will need:\n",
    "\n",
    "1. Azure OpenAI access and a resource with a chat model deployed (for example, GPT-3 or GPT-4)\n",
    "2. Azure Cognitive Search resource\n",
    "3. Azure Blob Storage resource\n",
    "4. Your documents to be used as data (See [data source options](https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data#data-source-options))\n",
    "\n",
    "\n",
    "For a full walk-through on how to upload your documents to blob storage and create an index using the Azure AI Studio, see this [Quickstart](https://learn.microsoft.com/azure/ai-services/openai/use-your-data-quickstart?pivots=programming-language-studio&tabs=command-line)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First, we install the necessary dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install \"openai>=0.28.1,<1.0.0\"\n",
    "! pip install python-dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we'll use `dotenv` to load our environment variables. To connect with Azure OpenAI and the Search index, the following variables should be added to a `.env` file in `KEY=VALUE` format:\n",
    "\n",
    "* `OPENAI_API_BASE` - the Azure OpenAI endpoint. This can be found under \"Keys and Endpoints\" for your Azure OpenAI resource in the Azure Portal.\n",
    "* `OPENAI_API_KEY` - the Azure OpenAI API key. This can be found under \"Keys and Endpoints\" for your Azure OpenAI resource in the Azure Portal. Omit if using Azure Active Directory authentication (see below `Authentication using Microsoft Active Directory`)\n",
    "* `SEARCH_ENDPOINT` - the Cognitive Search endpoint. This URL be found on the \"Overview\" of your Search resource on the Azure Portal.\n",
    "* `SEARCH_KEY` - the Cognitive Search API key. Found under \"Keys\" for your Search resource in the Azure Portal.\n",
    "* `SEARCH_INDEX_NAME` - the name of the index you created with your own data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "import dotenv\n",
    "\n",
    "dotenv.load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_base = os.environ[\"OPENAI_API_BASE\"]\n",
    "\n",
    "# Azure OpenAI on your own data is only supported by the 2023-08-01-preview API version\n",
    "openai.api_version = \"2023-08-01-preview\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authentication\n",
    "\n",
    "The Azure OpenAI service supports multiple authentication mechanisms that include API keys and Azure credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_azure_active_directory = False  # Set this flag to True if you are using Azure Active Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Authentication using API key\n",
    "\n",
    "To set up the OpenAI SDK to use an *Azure API Key*, we need to set up the `api_type` to `azure` and set `api_key` to a key associated with your endpoint (you can find this key in *\"Keys and Endpoints\"* under *\"Resource Management\"* in the [Azure Portal](https://portal.azure.com))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not use_azure_active_directory:\n",
    "    openai.api_type = 'azure'\n",
    "    openai.api_key = os.environ[\"OPENAI_API_KEY\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Authentication using Microsoft Active Directory\n",
    "Let's now see how we can get a key via Microsoft Active Directory Authentication. See the [documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/managed-identity) for more information on how to set this up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install azure-identity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "if use_azure_active_directory:\n",
    "    default_credential = DefaultAzureCredential()\n",
    "    token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n",
    "\n",
    "    openai.api_type = \"azure_ad\"\n",
    "    openai.api_key = token.token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [


{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Azure functions example\n",
    "\n",
    "> Note: There is a newer version of the openai library available. See https://github.com/openai/openai-python/discussions/742\n",
    "\n",
    "This notebook shows how to use the function calling capability with the Azure OpenAI service. Functions allow a caller of chat completions to define capabilities that the model can use to extend its\n",
    "functionality into external tools and data sources.\n",
    "\n",
    "You can read more about chat functions on OpenAI's blog: https://openai.com/blog/function-calling-and-other-api-updates\n",
    "\n",
    "**NOTE**: Chat functions require model versions beginning with gpt-4 and gpt-35-turbo's `-0613` labels. They are not supported by older versions of the models."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First, we install the necessary dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install \"openai>=0.28.1,<1.0.0\"\n",
    "# (Optional) If you want to use Microsoft Active Directory\n",
    "! pip install azure-identity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Additionally, to properly access the Azure OpenAI Service, we need to create the proper resources at the [Azure Portal](https://portal.azure.com) (you can check a detailed guide on how to do this in the [Microsoft Docs](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal))\n",
    "\n",
    "Once the resource is created, the first thing we need to use is its endpoint. You can get the endpoint by looking at the *\"Keys and Endpoints\"* section under the *\"Resource Management\"* section. Having this, we will set up the SDK using this information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_base = \"\" # Add your endpoint here\n",
    "\n",
    "# functions is only supported by the 2023-07-01-preview API version\n",
    "openai.api_version = \"2023-07-01-preview\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authentication\n",
    "\n",
    "The Azure OpenAI service supports multiple authentication mechanisms that include API keys and Azure credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_azure_active_directory = False"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Authentication using API key\n",
    "\n",
    "To set up the OpenAI SDK to use an *Azure API Key*, we need to set up the `api_type` to `azure` and set `api_key` to a key associated with your endpoint (you can find this key in *\"Keys and Endpoints\"* under *\"Resource Management\"* in the [Azure Portal](https://portal.azure.com))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not use_azure_active_directory:\n",
    "    openai.api_type = \"azure\"\n",
    "    openai.api_key = os.environ[\"OPENAI_API_KEY\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: In this example, we configured the library to use the Azure API by setting the variables in code. For development, consider setting the environment variables instead:\n",
    "\n",
    "```\n",
    "OPENAI_API_BASE\n",
    "OPENAI_API_KEY\n",
    "OPENAI_API_TYPE\n",
    "OPENAI_API_VERSION\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Authentication using Microsoft Active Directory\n",
    "Let's now see how we can get a key via Microsoft Active Directory Authentication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "if use_azure_active_directory:\n",
    "    default_credential = DefaultAzureCredential()\n",
    "    token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n",
    "\n",
    "    openai.api_type = \"azure_ad\"\n",
    "    openai.api_key = token.token"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A token is valid for a period of time, after which it will expire. To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import typing\n",
    "import time\n",
    "import requests\n",
    "\n",
    "if typing.TYPE_CHECKING:\n",
    "    from azure.core.credentials import TokenCredential\n",
    "\n",
    "class TokenRefresh(requests.auth.AuthBase):\n",
    "\n",
    "    def __init__(self, credential: \"TokenCredential\", scopes: typing.List[str]) -> None:\n",
    "        self.credential = credential\n",
    "        self.scopes = scopes\n",
    "        self.cached_token: typing.Optional[str] = None\n",
    "\n",
    "    def __call__(self, req):\n",
    "        if not self.cached_token or self.cached_token.expires_on - time.time() < 300:\n",
    "            self.cached_token = self.credential.get_token(*self.scopes)\n",
    "        req.headers[\"Authorization\"] = f\"Bearer {self.cached_token.token}\"\n",
    "        return req\n",
    "\n",
    "if use_azure_active_directory:\n",
    "    session = requests.Session()\n",
    "    session.auth = TokenRefresh(default_credential, [\"https://cognitiveservices.azure.com/.default\"])\n",
    "\n",
    "    openai.requestssession = session"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions\n",
    "\n",
    "With setup and authentication complete, you can now use functions with the Azure OpenAI service. This will be split into a few steps:\n",
    "\n",
    "1. Define the function(s)\n",
    "2. Pass function definition(s) into chat completions API\n",
    "3. Call function with arguments from the response\n",
    "4. Feed function response back into chat completions API"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Define the function(s)\n",
    "\n",
    "A list of functions can be defined, each containing the name of the function, an optional description, and the parameters the function accepts (described as a JSON schema)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "functions = [\n",
    "    {\n",
    "        \"name\": \"get_current_weather\",\n",
    "        \"description\": \"Get the current weather\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"location\": {\n",
    "                    \"type\": \"string\",\n",
    "                    \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
    "                },\n",


"        {\"role\": \"assistant\", \"content\": \"Who's there?\"},\n",
    "        {\"role\": \"user\", \"content\": \"Orange.\"},\n",
    "    ],\n",
    "    temperature=0,\n",
    "    stream=True\n",
    ")\n",
    "\n",
    "for chunk in response:\n",
    "    if len(chunk.choices) > 0:\n",
    "        delta = chunk.choices[0].delta\n",
    "\n",
    "        if \"role\" in delta.keys():\n",
    "            print(delta.role + \": \", end=\"\", flush=True)\n",
    "        if \"content\" in delta.keys():\n",
    "            print(delta.content, end=\"\", flush=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}


{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# How to use GPT-4 Vision with Function Calling \n",
    "\n",
    "The new GPT-4 Turbo model, available as gpt-4-turbo-2024-04-09 as of April 2024, now enables function calling with vision capabilities, better reasoning and a knowledge cutoff date of Dec 2023. Using images with function calling will unlock multimodal use cases and the ability to use reasoning, allowing you to go beyond OCR and image descriptions.\n",
    "\n",
    "We will go through two examples to demonstrate the use of function calling with GPT-4 Turbo with Vision:\n",
    "\n",
    "1. Simulating a customer service assistant for delivery exception support\n",
    "2. Analyzing an organizational chart to extract employee information"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "7b3d17451440ca82"
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Installation and Setup"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "feffa794bc28be22"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "!pip install pymupdf --quiet\n",
    "!pip install openai --quiet\n",
    "!pip install matplotlib --quiet\n",
    "# instructor makes it easy to work with function calling\n",
    "!pip install instructor --quiet"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "6ce24dfd4e4bbc47",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "initial_id",
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2024-04-10T03:50:37.564145Z",
     "start_time": "2024-04-10T03:50:37.560040Z"
    }
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "from enum import Enum\n",
    "from io import BytesIO\n",
    "from typing import Iterable\n",
    "from typing import List\n",
    "from typing import Literal, Optional\n",
    "\n",
    "import fitz\n",
    "# Instructor is powered by Pydantic, which is powered by type hints. Schema validation, prompting is controlled by type annotations\n",
    "import instructor\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "from PIL import Image\n",
    "from openai import OpenAI\n",
    "from pydantic import BaseModel, Field"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 1. Simulating a customer service assistant for delivery exception support\n",
    "We will simulate a customer service assistant for a delivery service that is equipped to analyze images of packages. The assistant will perform the following actions based on the image analysis:\n",
    "- If a package appears damaged in the image, automatically process a refund according to policy.\n",
    "- If the package looks wet, initiate a replacement.\n",
    "- If the package appears normal and not damaged, escalate to an agent."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "14ab8f931cc759d"
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's look at the sample images of packages that the customer service assistant will analyze to determine the appropriate action. We will encode the images as base64 strings for processing by the model."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "ca35604f27f93e77"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded image: wet_package.jpg\n",
      "Encoded image: damaged_package.jpg\n",
      "Encoded image: normal_package.jpg\n"
     ]
    },
    {
     "data": {
      "text/plain": "<Figure size 1800x600 with 3 Axes>",
      ""
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Function to encode the image as base64\n",
    "def encode_image(image_path: str):\n",
    "    # check if the image exists\n",
    "    if not os.path.exists(image_path):\n",
    "        raise FileNotFoundError(f\"Image file not found: {image_path}\")\n",
    "    with open(image_path, \"rb\") as image_file:\n",
    "        return base64.b64encode(image_file.read()).decode('utf-8')\n",
    "\n",
    "\n",
    "# Sample images for testing\n",
    "image_dir = \"images\"\n",
    "\n",
    "# encode all images within the directory\n",
    "image_files = os.listdir(image_dir)\n",
    "image_data = {}\n",
    "for image_file in image_files:\n",
    "    image_path = os.path.join(image_dir, image_file)\n",
    "    # encode the image with key as the image file name\n",
    "    image_data[image_file.split('.')[0]] = encode_image(image_path)\n",
    "    print(f\"Encoded image: {image_file}\")\n",
    "\n",
    "\n",
    "def display_images(image_data: dict):\n",
    "    fig, axs = plt.subplots(1, 3, figsize=(18, 6))\n",
    "    for i, (key, value) in enumerate(image_data.items()):\n",
    "        img = Image.open(BytesIO(base64.b64decode(value)))\n",
    "        ax = axs[i]\n",
    "        ax.imshow(img)\n",
    "        ax.axis(\"off\")\n",
    "        ax.set_title(key)\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "display_images(image_data)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-04-10T03:50:38.120670Z",
     "start_time": "2024-04-10T03:50:37.565555Z"
    }
   },
   "id": "2f2066d40dbbfbe8",
   "execution_count": 45
  },
  {
   "cell_type": "markdown",
   "source": [
    "We have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.\n",
    "\n",
    "Let's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.\n"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "f4b16c5219dff6db"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "MODEL = \"gpt-4-turbo-2024-04-09\"\n",
    "\n",
    "class Order(BaseModel):\n",
    "    \"\"\"Represents an order with details such as order ID, customer name, product name, price, status, and delivery date.\"\"\"\n",
    "    order_id: str = Field(..., description=\"The unique identifier of the order\")\n",
    "    product_name: str = Field(..., description=\"The name of the product\")\n",
    "    price: float = Field(..., description=\"The price of the product\")\n",
    "    status: str = Field(..., description=\"The status of the order\")\n",
    "    delivery_date: str = Field(..., description=\"The delivery date of the order\")\n",
    "# Placeholder functions for order processing\n",
    "\n",
    "def get_order_details(order_id):\n",
    "    # Placeholder function to retrieve order details based on the order ID\n",
    "    return Order(\n",
    "        order_id=order_id,\n",
    "        product_name=\"Product X\",\n",
    "        price=100.0,\n",
    "        status=\"Delivered\",\n",
    "        delivery_date=\"2024-04-10\",\n",
    "    )\n",
    "\n",
    "def escalate_to_agent(order: Order, message: str):\n",
    "    # Placeholder function to escalate the order to a human agent\n",


"      1. Thank the customer and ask for clarification:
   a. \"Thank you for reaching out! Could you please specify whether you require a Business Associate Agreement (BAA) for using our API or for ChatGPT Enterprise?\"

2. If the customer requires a BAA for the API, then:
   a. Inform the customer: \"To obtain a BAA for our API, please email baa@openai.com with details about your company and use case.\"
   b. Inform the customer: \"Our team will respond within 1-2 business days.\"
   c. Inform the customer: \"We review each BAA request on a case-by-case basis and may need additional information.\"
   d. Inform the customer: \"The process is usually completed within a few business days.\"
   e. Inform the customer: \"Please note that only endpoints eligible for zero data retention are covered by the BAA.\"
      i. Call the `provide_list_of_zero_retention_endpoints` function.
   f. Inform the customer: \"An enterprise agreement is not required to sign a BAA.\"

3. If the customer requires a BAA for ChatGPT Enterprise, then:
   a. Inform the customer: \"To explore a BAA for ChatGPT Enterprise, please contact our sales team.\"
      i. Call the `provide_sales_contact_information` function.

4. If the customer is not approved, then:
   a. Inform the customer: \"We are able to approve most customers that request BAAs, but occasionally a use case doesn't pass our team's evaluation.\"
   b. Inform the customer: \"In that case, we'll provide feedback and context as to why and give you the opportunity to update your intended use of our API and re-apply.\"

5. Ask the customer if there is anything else you can assist with:
   a. \"Is there anything else I can assist you with today?\"

6. Call the `case_resolution` function.

---

**Function Definitions:**

- `provide_list_of_zero_retention_endpoints`:
  - **Purpose**: Provides the customer with a list of API endpoints that are eligible for zero data retention under the BAA.
  - **Parameters**: None.

- `provide_sales_contact_information`:
  - **Purpose**: Provides the customer with contact information to reach our sales team for ChatGPT Enterprise inquiries.
  - **Parameters**: None.

- `case_resolution`:
  - **Purpose**: Finalizes the case and marks it as resolved.
  - **Parameters**: None.	\n",
       "
\n",
       "
\n",
       "      2	\n",
       "      Set up prepaid billing	\n",
       "      How can I set up prepaid billing?

How it works
Prepaid billing allows API users to pre-purchase usage. The credits you've bought will be applied to your monthly invoice. This means that any API usage you incur will first be deducted from the prepaid credits. If your usage exceeds the credits you've purchased, you'll then be billed for the additional amount.
Prepaid billing helps developers know what they are committing to upfront which can provide more predictability for budgeting and spend management.

Setting up prepaid billing
If you're on a Monthly Billing plan, you may also choose to switch to prepaid billing and purchase credits upfront for API usage.
- Go to your billing overview in your account settings
- Click \"Start payment plan\" (you may see variations like \"Buy credits\")
    Note: If you previously had an arrears billing plan, you'll need to cancel this existing payment plan first.
- Choose the initial amount of credits you want to purchase. The minimum purchase is $5. The maximum purchase will be based on your trust tier.
- Confirm and purchase your initial amount of credits.
- Use auto-recharge to set an automatic recharge amount, which is the amount of credits that will be added to your account when your balance falls below a set threshold.

Please note that any purchased credits will expire after 1 year and they are non-refundable.
After you’ve purchased credits, you should be able to start using the API. Note that there may be a couple minutes of delay while our systems update to reflect your credit balance.

Purchasing additional credits
Once you’ve consumed all your credits, your API requests will start returning an error letting you know you’ve hit your billing quota. If you’d like to continue your API usage, you can return to the billing portal and use the “Add to balance” button to purchase additional credits.

Delayed billing
Due to the complexity of our billing and processing systems, there may be delays in our ability to cut off access after you consume all of your credits. This excess usage may appear as a negative credit balance in your billing dashboard, and will be deducted from your next credit purchase.
	\n",
       "      1. `call check_billing_plan(user_id)`
   - **Function:** `check_billing_plan(user_id)`
     - **Purpose:** Retrieves the user's current billing plan (e.g., Monthly Billing, Prepaid Billing, or Arrears Billing).
     - **Parameters:**
       - `user_id`: The unique identifier of the user.

2. If the user has an arrears billing plan:
   2a. Inform the user: \"Please note that since you have an arrears billing plan, you'll need to cancel your existing payment plan before switching to prepaid billing. Would you like assistance with cancelling your current plan?\"
   2b. If the user agrees, `call cancel_payment_plan(user_id)`
       - **Function:** `cancel_payment_plan(user_id)`
         - **Purpose:** Cancels the user's current arrears billing plan.
         - **Parameters:**
           - `user_id`: The unique identifier of the user.

3. Guide the user to set up prepaid billing:
   3a. Instruct the user: \"Please go to your billing overview in your account settings.\"
   3b. Instruct the user: \"Click on 'Start payment plan' (you may see variations like 'Buy credits').\"
   3c. Inform the user: \"Choose the initial amount of credits you want to purchase. The minimum purchase is $5, and the maximum purchase will be based on your trust tier.\"
   3d. Instruct the user: \"Confirm and purchase your initial amount of credits.\"
   3e. Suggest to the user: \"You can set up auto-recharge to automatically add credits to your account when your balance falls below a set threshold.\"

4. Inform the user about credit expiration and refund policy:
   4a. Inform the user: \"Please note that any purchased credits will expire after 1 year and they are non-refundable.\"

5. Inform the user about activation time:
   5a. Inform the user: \"After you’ve purchased credits, you should be able to start using the API. Note that there may be a couple of minutes delay while our systems update to reflect your credit balance.\"

6. Ask the user: \"Is there anything else I can assist you with today?\"

7. If the user has no further questions, `call case_resolution()`
   - **Function:** `case_resolution()`
     - **Purpose:** Marks the case as resolved and ends the interaction.	\n",
       "
\n",
       "
\n",
       "      3	\n",
       "      VAT Exemption request	\n",


"        try:\n",
                "            result = future.result()\n",
                "            data.extend(result)\n",
                "        except Exception as e:\n",
                "            print(f\"Error processing file: {str(e)}\")\n",
                "\n",
                "# Write the data to a CSV file\n",
                "csv_file = os.path.join(\"..\", \"embedded_data.csv\")\n",
                "with open(csv_file, 'w', newline='', encoding='utf-8') as csvfile:\n",
                "    fieldnames = [\"id\", \"vector_id\", \"title\", \"text\", \"title_vector\", \"content_vector\",\"category\"]\n",
                "    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
                "    writer.writeheader()\n",
                "    for row in data:\n",
                "        writer.writerow(row)\n",
                "        print(f\"Wrote row with id {row['id']} to CSV\")\n",
                "\n",
                "# Convert the CSV file to a Dataframe\n",
                "article_df = pd.read_csv(\"../embedded_data.csv\")\n",
                "# Read vectors from strings back into a list using json.loads\n",
                "article_df[\"title_vector\"] = article_df.title_vector.apply(json.loads)\n",
                "article_df[\"content_vector\"] = article_df.content_vector.apply(json.loads)\n",
                "article_df[\"vector_id\"] = article_df[\"vector_id\"].apply(str)\n",
                "article_df[\"category\"] = article_df[\"category\"].apply(str)\n",
                "article_df.head()\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We now have an `embedded_data.csv` file with six columns that we can upload to our vector database! "
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# Create Azure AI Vector Search"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "##  Create index\n",
                "We'll define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and hybrid search capabilities. For more details, visit Microsoft's documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "index_name = \"azure-ai-search-openai-cookbook-demo\"\n",
                "# index_name = \"<insert_name_for_index>\"\n",
                "\n",
                "index_client = SearchIndexClient(\n",
                "    endpoint=search_service_endpoint, credential=AzureKeyCredential(search_service_api_key)\n",
                ")\n",
                "# Define the fields for the index. Update these based on your data.\n",
                "# Each field represents a column in the search index\n",
                "fields = [\n",
                "    SimpleField(name=\"id\", type=SearchFieldDataType.String),  # Simple string field for document ID\n",
                "    SimpleField(name=\"vector_id\", type=SearchFieldDataType.String, key=True),  # Key field for the index\n",
                "    # SimpleField(name=\"url\", type=SearchFieldDataType.String),  # URL field (commented out)\n",
                "    SearchableField(name=\"title\", type=SearchFieldDataType.String),  # Searchable field for document title\n",
                "    SearchableField(name=\"text\", type=SearchFieldDataType.String),  # Searchable field for document text\n",
                "    SearchField(\n",
                "        name=\"title_vector\",\n",
                "        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),  # Collection of single values for title vector\n",
                "        vector_search_dimensions=1536,  # Number of dimensions in the vector\n",
                "        vector_search_profile_name=\"my-vector-config\",  # Profile name for vector search configuration\n",
                "    ),\n",
                "    SearchField(\n",
                "        name=\"content_vector\",\n",
                "        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),  # Collection of single values for content vector\n",
                "        vector_search_dimensions=1536,  # Number of dimensions in the vector\n",
                "        vector_search_profile_name=\"my-vector-config\",  # Profile name for vector search configuration\n",
                "    ),\n",
                "    SearchableField(name=\"category\", type=SearchFieldDataType.String, filterable=True),  # Searchable field for document category\n",
                "]\n",
                "\n",
                "# This configuration defines the algorithm and parameters for vector search\n",
                "vector_search = VectorSearch(\n",
                "    algorithms=[\n",
                "        HnswAlgorithmConfiguration(\n",
                "            name=\"my-hnsw\",  # Name of the HNSW algorithm configuration\n",
                "            kind=VectorSearchAlgorithmKind.HNSW,  # Type of algorithm\n",
                "            parameters=HnswParameters(\n",
                "                m=4,  # Number of bi-directional links created for every new element\n",
                "                ef_construction=400,  # Size of the dynamic list for the nearest neighbors during construction\n",
                "                ef_search=500,  # Size of the dynamic list for the nearest neighbors during search\n",
                "                metric=VectorSearchAlgorithmMetric.COSINE,  # Distance metric used for the search\n",
                "            ),\n",
                "        )\n",
                "    ],\n",
                "    profiles=[\n",
                "        VectorSearchProfile(\n",
                "            name=\"my-vector-config\",  # Name of the vector search profile\n",
                "            algorithm_configuration_name=\"my-hnsw\",  # Reference to the algorithm configuration\n",
                "        )\n",
                "    ],\n",
                ")\n",
                "\n",
                "# Create the search index with the vector search configuration\n",
                "# This combines all the configurations into a single search index\n",
                "index = SearchIndex(\n",
                "    name=index_name,  # Name of the index\n",
                "    fields=fields,  # Fields defined for the index\n",
                "    vector_search=vector_search  # Vector search configuration\n",
                "\n",
                ")\n",
                "\n",
                "# Create or update the index\n",
                "# This sends the index definition to the Azure Search service\n",
                "result = index_client.create_index(index)\n",
                "print(f\"{result.name} created\")  # Output the name of the created index"
            ]
        },
        {
            "attachments": {},
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Upload Data\n",
                "\n",
                "Now we'll upload the articles from above that we've stored in `embedded_data.csv` from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to  [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import).\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Convert the 'id' and 'vector_id' columns to string so one of them can serve as our key field\n",
                "article_df[\"id\"] = article_df[\"id\"].astype(str)\n",
                "article_df[\"vector_id\"] = article_df[\"vector_id\"].astype(str)\n",
                "\n",
                "# Convert the DataFrame to a list of dictionaries\n",
                "documents = article_df.to_dict(orient=\"records\")\n",
                "\n",
                "# Log the number of documents to be uploaded\n",
                "print(f\"Number of documents to upload: {len(documents)}\")\n",
                "\n",
                "# Create a SearchIndexingBufferedSender\n",
                "batch_client = SearchIndexingBufferedSender(\n",
                "    search_service_endpoint, index_name, AzureKeyCredential(search_service_api_key)\n",
                ")\n",
                "# Get the first document to check its schema\n",
                "first_document = documents[0]\n",
                "\n",
                "# Get the index schema\n",
                "index_schema = index_client.get_index(index_name)\n",
                "\n",
                "# Get the field names from the index schema\n",
                "index_fields = {field.name: field.type for field in index_schema.fields}\n",
                "\n",
                "# Check each field in the first document\n",
                "for field, value in first_document.items():\n",
                "    if field not in index_fields:\n",


const { Client } = require('@microsoft/microsoft-graph-client');
const pdfParse = require('pdf-parse');
const { Buffer } = require('buffer');
const path = require('path');
const axios = require('axios');
const qs = require('querystring');
const { OpenAI } = require("openai");

//// --------- ENVIRONMENT CONFIGURATION AND INITIALIZATION ---------
// Function to initialize Microsoft Graph client
const initGraphClient = (accessToken) => {
    return Client.init({
        authProvider: (done) => {
            done(null, accessToken); // Pass the access token for Graph API calls
        }
    });
};

//// --------- AUTHENTICATION AND TOKEN MANAGEMENT ---------
// Function to obtain OBO token. This will take the access token in request header (scoped to this Function App) and generate a new token to use for Graph API
const getOboToken = async (userAccessToken) => {
    const { TENANT_ID, CLIENT_ID,  MICROSOFT_PROVIDER_AUTHENTICATION_SECRET } = process.env;
    const scope = 'https://graph.microsoft.com/.default';
    const oboTokenUrl = `https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token`;

    const params = {
        client_id: CLIENT_ID,
        client_secret: MICROSOFT_PROVIDER_AUTHENTICATION_SECRET,
        grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
        assertion: userAccessToken,
        requested_token_use: 'on_behalf_of',
        scope: scope
    };

    try {
        const response = await axios.post(oboTokenUrl, qs.stringify(params), {
            headers: {
                'Content-Type': 'application/x-www-form-urlencoded'
            }
        });
        return response.data.access_token; // OBO token
    } catch (error) {
        console.error('Error obtaining OBO token:', error.response?.data || error.message);
        throw error;
    }
};
//// --------- DOCUMENT PROCESSING ---------
// Function to fetch drive item content and convert to text
const getDriveItemContent = async (client, driveId, itemId, name) => {
    try {
        const fileType = path.extname(name).toLowerCase();
        // the below files types are the ones that are able to be converted to PDF to extract the text. See https://learn.microsoft.com/en-us/graph/api/driveitem-get-content-format?view=graph-rest-1.0&tabs=http
        const allowedFileTypes = ['.pdf', '.doc', '.docx', '.odp', '.ods', '.odt', '.pot', '.potm', '.potx', '.pps', '.ppsx', '.ppsxm', '.ppt', '.pptm', '.pptx', '.rtf'];
        // filePath changes based on file type, adding ?format=pdf to convert non-pdf types to pdf for text extraction, so all files in allowedFileTypes above are converted to pdf
        const filePath = `/drives/${driveId}/items/${itemId}/content` + ((fileType === '.pdf' || fileType === '.txt' || fileType === '.csv') ? '' : '?format=pdf');
        if (allowedFileTypes.includes(fileType)) {
            response = await client.api(filePath).getStream();
            // The below takes the chunks in response and combines
            let chunks = [];
            for await (let chunk of response) {
                chunks.push(chunk);
            }
            let buffer = Buffer.concat(chunks);
            // the below extracts the text from the PDF.
            const pdfContents = await pdfParse(buffer);
            return pdfContents.text;
        } else if (fileType === '.txt') {
            // If the type is txt, it does not need to create a stream and instead just grabs the content
            response = await client.api(filePath).get();
            return response;
        }  else if (fileType === '.csv') {
            response = await client.api(filePath).getStream();
            let chunks = [];
            for await (let chunk of response) {
                chunks.push(chunk);
            }
            let buffer = Buffer.concat(chunks);
            let dataString = buffer.toString('utf-8');
            return dataString

    } else {
        return 'Unsupported File Type';
    }

    } catch (error) {
        console.error('Error fetching drive content:', error);
        throw new Error(`Failed to fetch content for ${name}: ${error.message}`);
    }
};

// Function to get relevant parts of text using got-4o-mini.
const getRelevantParts = async (text, query) => {
    try {
        // We use your OpenAI key to initialize the OpenAI client
        const openAIKey = process.env["OPENAI_API_KEY"];
        const openai = new OpenAI({
            apiKey: openAIKey,
        });
        const response = await openai.chat.completions.create({
            // Using gpt-4o-mini due to speed to prevent timeouts. You can tweak this prompt as needed
            model: "gpt-4o-mini",
            messages: [
                {"role": "system", "content": "You are a helpful assistant that finds relevant content in text based on a query. You only return the relevant sentences, and you return a maximum of 10 sentences"},
                {"role": "user", "content": `Based on this question: **"${query}"**, get the relevant parts from the following text:*****\n\n${text}*****. If you cannot answer the question based on the text, respond with 'No information provided'`}
            ],
            // using temperature of 0 since we want to just extract the relevant content
            temperature: 0,
            // using max_tokens of 1000, but you can customize this based on the number of documents you are searching.
            max_tokens: 1000
        });
        return response.choices[0].message.content;
    } catch (error) {
        console.error('Error with OpenAI:', error);
        return 'Error processing text with OpenAI' + error;
    }
};

//// --------- AZURE FUNCTION LOGIC ---------
// Below is what the Azure Function executes


"text": "RAG\nTechnique\n\nFebruary 2024\n\n\fOverview\n\nRetrieval-Augmented Generation \nenhances the capabilities of language \nmodels by combining them with a \nretrieval system. This allows the model \nto leverage external knowledge sources \nto generate more accurate and \ncontextually relevant responses.\n\nExample use cases\n\n- Provide answers with up-to-date \n\ninformation\n\n- Generate contextual responses\n\nWhat we\u2019ll cover\n\n\u25cf Technical patterns\n\n\u25cf Best practices\n\n\u25cf Common pitfalls\n\n\u25cf Resources\n\n3\n\n\fWhat is RAG\n\nRetrieve information to Augment the model\u2019s knowledge and Generate the output\n\n\u201cWhat is your \nreturn policy?\u201d\n\nask\n\nresult\n\nsearch\n\nLLM\n\nreturn information\n\nTotal refunds: 0-14 days\n50% of value vouchers: 14-30 days\n$5 discount on next order: > 30 days\n\n\u201cYou can get a full refund up \nto 14 days after the \npurchase, then up to 30 days \nyou would get a voucher for \nhalf the value of your order\u201d\n\nKnowledge \nBase / External \nsources\n\n4\n\n\fWhen to use RAG\n\nGood for  \u2705\n\nNot good for  \u274c\n\n\u25cf\n\n\u25cf\n\nIntroducing new information to the model \n\n\u25cf\n\nTeaching the model a speci\ufb01c format, style, \n\nto update its knowledge\n\nReducing hallucinations by controlling \n\ncontent\n\n/!\\ Hallucinations can still happen with RAG\n\nor language\n\u2794 Use \ufb01ne-tuning or custom models instead\n\n\u25cf\n\nReducing token usage\n\u2794 Consider \ufb01ne-tuning depending on the use \n\ncase\n\n5\n\n\fTechnical patterns\n\nData preparation\n\nInput processing\n\nRetrieval\n\nAnswer Generation\n\n\u25cf Chunking\n\n\u25cf\n\n\u25cf\n\nEmbeddings\n\nAugmenting \ncontent\n\n\u25cf\n\nInput \naugmentation\n\n\u25cf NER\n\n\u25cf\n\nSearch\n\n\u25cf Context window\n\n\u25cf Multi-step \nretrieval\n\n\u25cf Optimisation\n\n\u25cf\n\nSafety checks\n\n\u25cf\n\nEmbeddings\n\n\u25cf Re-ranking\n\n6\n\n\fTechnical patterns\nData preparation\n\nchunk documents into multiple \npieces for easier consumption\n\ncontent\n\nembeddings\n\n0.983, 0.123, 0.289\u2026\n\n0.876, 0.145, 0.179\u2026\n\n0.983, 0.123, 0.289\u2026\n\nAugment content \nusing LLMs\n\nEx: parse text only, ask gpt-4 to rephrase & \nsummarize each part, generate bullet points\u2026\n\nBEST PRACTICES\n\nPre-process content for LLM \nconsumption: \nAdd summary, headers for each \npart, etc.\n+ curate relevant data sources\n\nKnowledge \nBase\n\nCOMMON PITFALLS\n\n\u2794 Having too much low-quality \n\ncontent\n\n\u2794 Having too large documents\n\n7\n\n\fTechnical patterns\nData preparation: chunking\n\nWhy chunking?\n\nIf your system doesn\u2019t require \nentire documents to provide \nrelevant answers, you can \nchunk them into multiple pieces \nfor easier consumption (reduced \ncost & latency).\n\nOther approaches: graphs or \nmap-reduce\n\nThings to consider\n\n\u25cf\n\nOverlap:\n\n\u25cb\n\n\u25cb\n\nShould chunks be independent or overlap one \nanother?\nIf they overlap, by how much?\n\n\u25cf\n\nSize of chunks: \n\n\u25cb What is the optimal chunk size for my use case?\n\u25cb\n\nDo I want to include a lot in the context window or \njust the minimum?\n\n\u25cf Where to chunk:\n\n\u25cb\n\n\u25cb\n\nShould I chunk every N tokens or use speci\ufb01c \nseparators? \nIs there a logical way to split the context that would \nhelp the retrieval process?\n\n\u25cf What to return:\n\n\u25cb\n\n\u25cb\n\nShould I return chunks across multiple documents \nor top chunks within the same doc?\nShould chunks be linked together with metadata to \nindicate common properties?\n\n8\n\n\fTechnical patterns\nData preparation: embeddings\n\nWhat to embed?\n\nDepending on your use case \nyou might not want just to \nembed the text in the \ndocuments but metadata as well \n- anything that will make it easier \nto surface this speci\ufb01c chunk or \ndocument when performing a \nsearch\n\nExamples\n\nEmbedding Q&A posts in a forum\nYou might want to embed the title of the posts, \nthe text of the original question and the content of \nthe top answers.\nAdditionally, if the posts are tagged by topic or \nwith keywords, you can embed those too.\n\nEmbedding product specs\nIn additional to embedding the text contained in \ndocuments describing the products, you might \nwant to add metadata that you have on the \nproduct such as the color, size, etc. in your \nembeddings.\n\n9\n\n\fTechnical patterns\nData preparation: augmenting content\n\nWhat does \u201cAugmenting \ncontent\u201d mean?\n\nAugmenting content refers to \nmodi\ufb01cations of the original content \nto make it more digestible for a \nsystem relying on RAG. The \nmodi\ufb01cations could be a change in \nformat, wording, or adding \ndescriptive content such as \nsummaries or keywords.\n\nExample approaches\n\nMake it a guide*\nReformat the content to look more like \na step-by-step guide with clear \nheadings and bullet-points, as this \nformat is more easily understandable \nby an LLM.\n\nAdd descriptive metadata*\nConsider adding keywords or text that \nusers might search for when thinking \nof a speci\ufb01c product or service.\n\nMultimodality\nLeverage models \nsuch as Whisper or \nGPT-4V to \ntransform audio or \nvisual content into \ntext.\nFor example, you \ncan use GPT-4V to \ngenerate tags for \nimages or to \ndescribe slides.\n\n* GPT-4 can do this for you with the right prompt\n\n10\n\n\fTechnical patterns\nInput processing\n\nProcess input according to task\n\nQ&A\nHyDE:  Ask LLM to hypothetically answer the \nquestion & use the answer to search the KB\n\nembeddings\n\n0.983, 0.123, 0.289\u2026\n\n0.876, 0.145, 0.179\u2026\n\nContent search\nPrompt LLM to rephrase input & optionally add \nmore context\n\nquery\n\nSELECT * from items\u2026\n\nDB search\nNER:  Find relevant entities to be used for a \nkeyword search or to construct a search query\n\nkeywords\n\nred\n\nsummer\n\nBEST PRACTICES\n\nConsider how to transform the \ninput to match content in the \ndatabase\nConsider using metadata to \naugment the user input\n\nCOMMON PITFALLS\n\n\u2794 Comparing directly the input \nto the database without \nconsidering the task \nspeci\ufb01cities \n\n11\n\n\fTechnical patterns\nInput processing: input augmentation\n\nWhat is input augmentation?\n\nExample approaches\n\nAugmenting the input means turning \nit into something di\ufb00erent, either \nrephrasing it, splitting it in several \ninputs or expanding it.\nThis helps boost performance as \nthe LLM might understand better \nthe user intent.\n\nQuery \nexpansion*\nRephrase the \nquery to be \nmore \ndescriptive\n\nHyDE*\nHypothetically \nanswer the \nquestion & use \nthe answer to \nsearch the KB\n\nSplitting a query in N*\nWhen there is more than 1 question or \nintent in a user query, consider \nsplitting it in several queries\n\nFallback\nConsider \n


implementing a \n\ufb02ow where the LLM \ncan ask for \nclari\ufb01cation when \nthere is not enough \ninformation in the \noriginal user query \nto get a result\n(Especially relevant \nwith tool usage)\n\n* GPT-4 can do this for you with the right prompt\n\n12\n\n\fTechnical patterns\nInput processing: NER\n\nWhy use NER?\n\nUsing NER (Named Entity \nRecognition) allows to extract \nrelevant entities from the input, that \ncan then be used for more \ndeterministic search queries. \nThis can be useful when the scope \nis very constrained.\n\nExample\n\nSearching for movies\nIf you have a structured database containing \nmetadata on movies, you can extract genre, \nactors or directors names, etc. from the user \nquery and use this to search the database\n\nNote: You can use exact values or embeddings after \nhaving extracted the relevant entities\n\n13\n\n\fTechnical patterns\nRetrieval\n\nre-ranking\n\nINPUT\n\nembeddings\n\n0.983, 0.123, 0.289\u2026\n\n0.876, 0.145, 0.179\u2026\n\nquery\n\nSELECT * from items\u2026\n\nkeywords\n\nred\n\nsummer\n\nSemantic \nsearch\n\nRESULTS\n\nRESULTS\n\nvector DB\n\nrelational / \nnosql db\n\nFINAL RESULT\n\nUsed to \ngenerate output\n\nBEST PRACTICES\n\nUse a combination of semantic \nsearch and deterministic queries \nwhere possible\n\n+ Cache output where possible\n\nCOMMON PITFALLS\n\n\u2794 The wrong elements could be \ncompared when looking at \ntext similarity, that is why \nre-ranking is important\n\n14\n\n\fTechnical patterns\nRetrieval: search\n\nHow to search?\n\nSemantic search\n\nKeyword search\n\nSearch query\n\nThere are many di\ufb00erent \napproaches to search depending on \nthe use case and the existing \nsystem.\n\nUsing embeddings, you \ncan perform semantic \nsearches. You can \ncompare embeddings \nwith what is in your \ndatabase and \ufb01nd the \nmost similar.\n\nIf you have extracted \nspeci\ufb01c entities or \nkeywords to search for, \nyou can search for these \nin your database.\n\nBased on the extracted \nentities you have or the \nuser input as is, you can \nconstruct search queries \n(SQL, cypher\u2026) and use \nthese queries to search \nyour database.\n\nYou can use a hybrid approach and combine several of these.\nYou can perform multiple searches in parallel or in sequence, or \nsearch for keywords with their embeddings for example.\n\n15\n\n\fTechnical patterns\nRetrieval: multi-step retrieval\n\nWhat is multi-step retrieval?\n\nIn some cases, there might be \nseveral actions to be performed to \nget the required information to \ngenerate an answer.\n\nThings to consider\n\n\u25cf\n\nFramework to be used:\n\n\u25cb When there are multiple steps to perform, \nconsider whether you want to handle this \nyourself or use a framework to make it easier\n\n\u25cf\n\nCost & Latency:\n\n\u25cb\n\n\u25cb\n\nPerforming multiple steps at the retrieval \nstage can increase latency and cost \nsigni\ufb01cantly\nConsider performing actions in parallel to \nreduce latency\n\n\u25cf\n\nChain of Thought:\n\n\u25cb\n\n\u25cb\n\nGuide the assistant with the chain of thought \napproach: break down instructions into \nseveral steps, with clear guidelines on \nwhether to continue, stop or do something \nelse. \nThis is more appropriate when tasks need to \nbe performed sequentially - for example: \u201cif \nthis didn\u2019t work, then do this\u201d\n\n16\n\n\fTechnical patterns\nRetrieval: re-ranking\n\nWhat is re-ranking?\n\nExample approaches\n\nRe-ranking means re-ordering the \nresults of the retrieval process to \nsurface more relevant results.\nThis is particularly important when \ndoing semantic searches.\n\nRule-based re-ranking\nYou can use metadata to rank results by relevance. For \nexample, you can look at the recency of the documents, at \ntags, speci\ufb01c keywords in the title, etc.\n\nRe-ranking algorithms\nThere are several existing algorithms/approaches you can use \nbased on your use case: BERT-based re-rankers, \ncross-encoder re-ranking, TF-IDF algorithms\u2026\n\n17\n\n\fTechnical patterns\nAnswer Generation\n\nFINAL RESULT\n\nPiece of content \nretrieved\n\nLLM\n\nPrompt including \nthe content\n\nUser sees the \n\ufb01nal result\n\nBEST PRACTICES\n\nEvaluate performance after each \nexperimentation to assess if it\u2019s \nworth exploring other paths\n+ Implement guardrails if applicable\n\nCOMMON PITFALLS\n\n\u2794 Going for \ufb01ne-tuning without \ntrying other approaches\n\u2794 Not paying attention to the \nway the model is prompted\n\n18\n\n\fTechnical patterns\nAnswer Generation: context window\n\nHow to manage context?\n\nDepending on your use case, there are \nseveral things to consider when \nincluding retrieved content into the \ncontext window to generate an answer. \n\nThings to consider\n\n\u25cf\n\nContext window max size:\n\n\u25cb\n\n\u25cb\n\nThere is a maximum size, so putting too \nmuch content is not ideal\nIn conversation use cases, the \nconversation will be part of the context \nas well and will add to that size\n\n\u25cf\n\nCost & Latency vs Accuracy:\n\n\u25cb More context results in increased \n\nlatency and additional costs since there \nwill be more input tokens\nLess context might also result in \ndecreased accuracy\n\n\u25cb\n\n\u25cf\n\n\u201cLost in the middle\u201d problem:\n\n\u25cb When there is too much context, LLMs \ntend to forget the text \u201cin the middle\u201d of \nthe content and might look over some \nimportant information.\n\n19\n\n\fTechnical patterns\nAnswer Generation: optimisation\n\nHow to optimise?\n\nThere are a few di\ufb00erent \nmethods to consider when \noptimising a RAG application.\nTry them from left to right, and \niterate with several of these \napproaches if needed.\n\nPrompt Engineering\n\nFew-shot examples\n\nFine-tuning\n\nAt each point of the \nprocess, experiment with \ndi\ufb00erent prompts to get \nthe expected input format \nor generate a relevant \noutput.\nTry guiding the model if \nthe process to get to the \n\ufb01nal outcome contains \nseveral steps.\n\nIf the model doesn\u2019t \nbehave as expected, \nprovide examples of what \nyou want e.g. provide \nexample user inputs and \nthe expected processing \nformat.\n\nIf giving a few examples \nisn\u2019t enough, consider \n\ufb01ne-tuning a model with \nmore examples for each \nstep of the process: you \ncan \ufb01ne-tune to get a \nspeci\ufb01c input processing \nor output format.\n\n20\n\n\fTechnical patterns\nAnswer Generation: safety checks\n\nWhy include safety checks?\n\nJust because you provide the model \nwith (supposedly) relevant context \ndoesn\u2019t mean the answer will \nsystematically be truthful or on-point.\nDepending on the use case, you \nmight want to double-check. \n\nExample evaluation framework: RAGAS\n\n21\n\n\f",


"pages_description": ["Overview\n\nRetrieval-Augmented Generation models enhance the capabilities of language models by combining them with a retrieval system. This allows the model to leverage external knowledge sources to generate more accurate and contextually relevant responses.\n\nExample use cases include providing answers with up-to-date information and generating contextual responses.\n\nWhat we'll cover includes technical patterns, best practices, common pitfalls, and resources.", "What is RAG\n\nThe content describes a process where a person asks a question, \"What is your return policy?\" This question is directed to an entity labeled LLM, which then searches a Knowledge Base or External sources. The Knowledge Base contains information on the return policy, stating that total refunds are available for 0-14 days, 50% of value in vouchers for 14-30 days, and a $5 discount on the next order for periods greater than 30 days. The LLM returns this information, and as a result, the person receives an answer: \"You can get a full refund up to 14 days after the purchase, then up to 30 days you would get a voucher for half the value of your order.\" The process illustrates how the RAG (Retrieve information to Augment the model's knowledge and Generate the output) system operates to provide answers to queries by retrieving relevant information from a knowledge source.", "When to use RAG\n\nThe content is divided into two sections, one highlighting the positive aspects of using RAG and the other outlining its limitations.\n\nGood for:\n- Introducing new information to the model to update its knowledge.\n- Reducing hallucinations by controlling content. However, it is noted that hallucinations can still happen with RAG.\n\nNot good for:\n- Teaching the model a specific format, style, or language. It is suggested to use fine-tuning or custom models instead.\n- Reducing token usage. For this purpose, fine-tuning should be considered depending on the use case.", "Technical patterns\n\nThe content outlines four key components of a technical process:\n\n1. Data preparation involves chunking, creating embeddings, and augmenting content.\n2. Input processing includes input augmentation, named entity recognition (NER), and the use of embeddings.\n3. Retrieval is characterized by search, multi-step retrieval, and re-ranking mechanisms.\n4. Answer Generation consists of establishing a context window, optimization, and performing safety checks.", "Technical patterns\nData preparation\n\nThe content describes a process for preparing data, specifically for chunking documents into multiple pieces to facilitate easier consumption. It involves converting content into embeddings, with numerical vectors representing the content, such as \"0.983, 0.123, 0.289...\" and so on. These embeddings are then used to populate a Knowledge Base.\n\nThere is a suggestion to augment content using Large Language Models (LLMs). For example, one could parse text only, ask GPT-4 to rephrase and summarize each part, and generate bullet points.\n\nBest practices are highlighted, emphasizing the need to pre-process content for LLM consumption by adding summaries, headers for each part, etc., and curating relevant data sources.\n\nCommon pitfalls are identified as having too much low-quality content and having too large documents.", "Technical patterns\nData preparation: chunking\n\nWhy chunking?\nChunking is discussed as a method for data preparation, where if a system does not require entire documents to provide relevant answers, documents can be chunked into multiple pieces for easier consumption, which results in reduced cost and latency. It is mentioned that other approaches include graphs or map-reduce.\n\nThings to consider\nSeveral considerations are listed for chunking:\n\n- Overlap: It is questioned whether chunks should be independent or overlap one another and, if they do overlap, by how much.\n- Size of chunks: The optimal chunk size for a specific use case is considered, as well as whether to include a lot in the context window or just the minimum.\n- Where to chunk: The discussion includes whether to chunk every N tokens or use specific separators and whether there is a logical way to split the context that would aid the retrieval process.\n- What to return: It is considered whether to return chunks across multiple documents or top chunks within the same document, and whether chunks should be linked together with metadata to indicate common properties.", "Technical patterns\nData preparation: embeddings\n\nWhat to embed?\nDepending on your use case you might not want just to embed the text in the documents but metadata as well - anything that will make it easier to surface this specific chunk or document when performing a search.\n\nExamples\nEmbedding Q&A posts in a forum\nYou might want to embed the title of the posts, the text of the original question and the content of the top answers. Additionally, if the posts are tagged by topic or with keywords, you can embed those too.\n\nEmbedding product specs\nIn addition to embedding the text contained in documents describing the products, you might want to add metadata that you have on the product such as the color, size, etc. in your embeddings.", "Technical patterns\nData preparation: augmenting content\n\nAugmenting content refers to modifications of the original content to make it more digestible for a system relying on RAG. The modifications could be a change in format, wording, or adding descriptive content such as summaries or keywords.\n\nExample approaches include:\n\n1. Make it a guide: Reformat the content to look more like a step-by-step guide with clear headings and bullet points, as this format is more easily understandable by an LLM.\n\n2. Add descriptive metadata: Consider adding keywords or text that users might search for when thinking of a specific product or service.\n\n3. Multimodality: Leverage models such as Whisper or GPT-4V to transform audio or visual content into text. For example, you can use GPT-4V to generate tags for images or to describe slides.\n\nNote: GPT-4 can assist with these tasks given the right prompt.", "Technical patterns: Input processing\n\nThe content describes various technical patterns for processing input data in relation to tasks. It outlines three specific approaches:\n\n1. Q&A: Utilize a hypothetical answer from a language model to search a knowledge base.\n2. Content search: Instruct a language model to rephrase input and possibly add more context.\n3. DB search: Employ Named Entity Recognition (NER) to identify relevant entities for keyword searches or to construct a search query.\n\nAdditionally, the content provides best practices and common pitfalls. Best practices include transforming the input to better match the content in the database and using metadata to enhance user input. A common pitfall to avoid is directly comparing the input to the database without considering the specificities of the task at hand.", "Technical patterns\nInput processing: input augmentation\n\nWhat is input augmentation?\n\nAugmenting the input means turning it into something different, either rephrasing it, splitting it in several inputs or expanding it. This helps boost performance as the LLM might understand better the user intent.\n\nExample approaches\n\nQuery expansion: Rephrase the query to be more descriptive.\n\nHyDE: Hypothetically answer the question & use the answer to search the KB.\n\nFallback: Consider implementing a flow where the LLM can ask for clarification when there is not enough information in the original user query to get a result (Especially relevant with tool usage).\n\nSplitting a query in N: When there is more than 1 question or intent in a user query, consider splitting it in several queries.\n\nNote: GPT-4 can do this for you with the right prompt.", "Technical patterns\n\nInput processing: NER\n\nWhy use NER?\n\nUsing NER (Named Entity Recognition) allows to extract relevant entities from the input, that can then be used for more deterministic search queries. This can be useful when the scope is very constrained.\n\nExample\n\nSearching for movies\n\nIf you have a structured database containing metadata on movies, you can extract genre, actors or directors names, etc. from the user query and use this to search the database.\n\nNote: You can use exact values or embeddings after having extracted the relevant entities.", "Technical patterns: Retrieval\n\nThe content describes a retrieval process involving various inputs and databases to produce a final result. The inputs include embeddings, which are numerical representations of data, and a query, exemplified by a SQL statement \"SELECT * from items...\". Additionally, keywords such as 'red' and 'summer' are used. These inputs interact with two types of databases: a vector database for semantic search and a relational or NoSQL database for keyword-based search.\n\nThe process involves searching these databases to retrieve initial results, which are then re-ranked to produce a refined set of results. The final result is used to generate output.\n\nBest practices highlighted include using a combination of semantic search and deterministic queries where possible, and caching output where feasible.\n\nCommon pitfalls mentioned involve the risk of comparing the wrong elements when looking at text similarity, emphasizing the importance of re-ranking in the retrieval process.",


function calling, and more.\nReturns a maximum of 4,096\n\noutput tokens. Learn more.\n\nhttps://platform.openai.com/docs/models/overview\n\n4/10\n\n\f26/02/2024, 17:58\n\nModels - OpenAI API\n\nMODEL\n\nDE S CRIPTION\n\ngpt-3.5-turbo-instruct Similar capabilities as GPT-3\nera models. Compatible with\nlegacy Completions endpoint\nand not Chat Completions.\n\nCONTEXT\nWIND OW\n\nTRAINING\nDATA\n\n4,096\ntokens\n\nUp to Sep\n2021\n\ngpt-3.5-turbo-16k\n\nLegacy  Currently points to\ngpt-3.5-turbo-16k-0613.\n\n16,385\ntokens\n\nUp to Sep\n2021\n\ngpt-3.5-turbo-0613\n\nLegacy  Snapshot of gpt-3.5-\n\nturbo from June 13th 2023.\n\nWill be deprecated on June 13,\n2024.\n\n4,096\ntokens\n\nUp to Sep\n2021\n\ngpt-3.5-turbo-16k-0613\n\nLegacy  Snapshot of gpt-3.5-\n\n16,385\n\nUp to Sep\n\n16k-turbo from June 13th\n\ntokens\n\n2021\n\n2023. Will be deprecated on\n\nJune 13, 2024.\n\nDALL\u00b7E\n\nDALL\u00b7E is a AI system that can create realistic images and art from a description in\n\nnatural language. DALL\u00b7E 3 currently supports the ability, given a prompt, to create a\n\nnew image with a specific size. DALL\u00b7E 2 also support the ability to edit an existing\n\nimage, or create variations of a user provided image.\n\nDALL\u00b7E 3 is available through our Images API along with DALL\u00b7E 2. You can try DALL\u00b7E 3\n\nthrough ChatGPT Plus.\n\nMODEL\n\nDE S CRIPTION\n\ndall-e-3\n\nNew  DALL\u00b7E 3\n\nThe latest DALL\u00b7E model released in Nov 2023. Learn more.\n\ndall-e-2 The previous DALL\u00b7E model released in Nov 2022. The 2nd iteration of\nDALL\u00b7E with more realistic, accurate, and 4x greater resolution images\nthan the original model.\n\nTTS\n\nTTS is an AI model that converts text to natural sounding spoken text. We offer two\ndifferent model variates,  tts-1  is optimized for real time text to speech use cases\nand  tts-1-hd  is optimized for quality. These models can be used with the Speech\n\nendpoint in the Audio API.\n\nhttps://platform.openai.com/docs/models/overview\n\n5/10\n\n\f26/02/2024, 17:58\n\nModels - OpenAI API\n\nMODEL\n\nDE S CRIPTION\n\ntts-1\n\nNew  Text-to-speech 1\nThe latest text to speech model, optimized for speed.\n\ntts-1-hd\n\nNew  Text-to-speech 1 HD\nThe latest text to speech model, optimized for quality.\n\nWhisper\n\nWhisper is a general-purpose speech recognition model. It is trained on a large dataset\nof diverse audio and is also a multi-task model that can perform multilingual speech\nrecognition as well as speech translation and language identification. The Whisper v2-\n\nlarge model is currently available through our API with the  whisper-1  model name.\n\nCurrently, there is no difference between the open source version of Whisper and the\n\nversion available through our API. However, through our API, we offer an optimized\ninference process which makes running Whisper through our API much faster than\n\ndoing it through other means. For more technical details on Whisper, you can read the\n\npaper.\n\nEmbeddings\n\nEmbeddings are a numerical representation of text that can be used to measure the\n\nrelatedness between two pieces of text. Embeddings are useful for search, clustering,\n\nrecommendations, anomaly detection, and classification tasks. You can read more\nabout our latest embedding models in the announcement blog post.\n\nMODEL\n\nDE S CRIPTION\n\ntext-embedding-\n3-large\n\nNew  Embedding V3 large\nMost capable embedding model for both\n\nenglish and non-english tasks\n\ntext-embedding-\n\nNew  Embedding V3 small\n\n3-small\n\nIncreased performance over 2nd generation ada\nembedding model\n\ntext-embedding-\nada-002\n\nMost capable 2nd generation embedding\nmodel, replacing 16 first generation models\n\nOUTP UT\nDIMENSION\n\n3,072\n\n1,536\n\n1,536\n\nModeration\n\nhttps://platform.openai.com/docs/models/overview\n\n6/10\n\n\f26/02/2024, 17:58\n\nModels - OpenAI API\n\nThe Moderation models are designed to check whether content complies with\nOpenAI's usage policies. The models provide classification capabilities that look for\ncontent in the following categories: hate, hate/threatening, self-harm, sexual,\nsexual/minors, violence, and violence/graphic. You can find out more in our moderation\n\nguide.\n\nModeration models take in an arbitrary sized input that is automatically broken up into\nchunks of 4,096 tokens. In cases where the input is more than 32,768 tokens,\n\ntruncation is used which in a rare condition may omit a small number of tokens from\nthe moderation check.\n\nThe final results from each request to the moderation endpoint shows the maximum\n\nvalue on a per category basis. For example, if one chunk of 4K tokens had a category\nscore of 0.9901 and the other had a score of 0.1901, the results would show 0.9901 in the\nAPI response since it is higher.\n\nMODEL\n\nDE S CRIPTION\n\nMAX\nTOKENS\n\ntext-moderation-latest Currently points to text-moderation-\n\n32,768\n\n007.\n\ntext-moderation-stable Currently points to text-moderation-\n\n32,768\n\n007.\n\ntext-moderation-007\n\nMost capable moderation model across\nall categories.\n\n32,768\n\nGPT base\n\nGPT base models can understand and generate natural language or code but are not\ntrained with instruction following. These models are made to be replacements for our\n\noriginal GPT-3 base models and use the legacy Completions API. Most customers\n\nshould use GPT-3.5 or GPT-4.\n\nMODEL\n\nDE S CRIPTION\n\nbabbage-002 Replacement for the GPT-3 ada and\n\nbabbage base models.\n\ndavinci-002 Replacement for the GPT-3 curie and\n\ndavinci base models.\n\nMAX\nTOKENS\n\nTRAINING\nDATA\n\n16,384\ntokens\n\n16,384\ntokens\n\nUp to Sep\n2021\n\nUp to Sep\n2021\n\nHow we use your data\n\nhttps://platform.openai.com/docs/models/overview\n\n7/10\n\n\f26/02/2024, 17:58\n\nModels - OpenAI API\n\nYour data is your data.\n\nAs of March 1, 2023, data sent to the OpenAI API will not be used to train or improve\n\nOpenAI models (unless you explicitly opt in). One advantage to opting in is that the\nmodels may get better at your use case over time.\n\nTo help identify abuse, API data may be retained for up to 30 days, after which it will be\n\ndeleted (unless otherwise required by law). For trusted customers with sensitive\napplications, zero data retention may be available. With zero data retention, request\nand response bodies are not persisted to any logging mechanism and exist only in\nmemory in order to serve the request.\n\nNote that this data policy does not apply to OpenAI's non-API consumer services like\nChatGPT or DALL\u00b7E Labs.\n\nDefault usage policies by endpoint\n\nENDP OINT\n\nDATA USED\nFOR TRAINING\n\nDEFAULT\nRETENTION\n\nELIGIBLE FOR\nZERO RETENTION\n\n/v1/chat/completions*\n\nNo\n\n30 days\n\nYes, except\n\nimage inputs*\n\n/v1/files\n\n/v1/assistants\n\n/v1/threads\n\n/v1/threads/messages\n\n


# RAG
### Technique

-----

## Overview

Retrieval-Augmented Generation
enhances the capabilities of language
models by combining them with a
retrieval system. This allows the model
to leverage external knowledge sources
to generate more accurate and
contextually relevant responses.

**Example use cases**

-  Provide answers with up-to-date
information

-  Generate contextual responses

#### What we’ll cover

-  Technical patterns

-  Best practices

-  Common pitfalls

-  Resources

-----

## What is RAG

**Retrieve information to Augment the model’s knowledge and Generate the output**

Total refunds: 0-14 days

“What is your 50% of value vouchers: 14-30 days

return policy?” $5 discount on next order: > 30 days

ask search

LLM

result return information

Knowledge

Base / External

“You can get a full refund up sources

to 14 days after the

purchase, then up to 30 days

you would get a voucher for

-----

## When to use RAG

###### Good for ✅

###### ✅

-  Introducing new information to the model

to update its knowledge

-  Reducing hallucinations by controlling

content

/!\ Hallucinations can still happen with RAG

###### ❌

Teaching the model a specific format, style,

or language

➔Use fine-tuning or custom models instead

Reducing token usage

➔Consider fine-tuning depending on the use

case

-----

## Technical patterns

**Data preparation**

-  Chunking

-  Embeddings

-  Augmenting
content

**Input processing**

-  Input
augmentation

-  NER

-  Embeddings

**Retrieval**

-  Search

-  Multi-step
retrieval

-  Re-ranking

**Answer Generation**

-  Context window

-  Optimisation

-  Safety checks

-----

## Technical patterns
##### Data preparation BEST PRACTICES

Pre-process content for LLM
consumption:

chunk documents into multiple Add summary, headers for each
pieces for easier consumption part, etc.

embeddings + curate relevant data sources

embeddings

0.983, 0.123, 0.289…

0.876, 0.145, 0.179…

content Knowledge **COMMON PITFALLS**

0.983, 0.123, 0.289…

➔ Having too much low-quality

Knowledge

Base

content

Augment content ➔ Having too large documents

|tent 0.983, 0.12|Col2|Col3|
|---|---|---|
|Augment content using LLMs|||
||Augment content using LLMs||
||||


E t t l k t 4 t h &

-----

## Technical patterns
##### Data preparation: chunking ● Overlap:

-  Should chunks be independent or overlap one
another?

-  If they overlap, by how much?

Size of chunks:

-  What is the optimal chunk size for my use case?

-  Do I want to include a lot in the context window or
just the minimum?

Where to chunk:

-  Should I chunk every N tokens or use specific
separators?

-  Is there a logical way to split the context that would
help the retrieval process?

What to return:

-  Should I return chunks across multiple documents
or top chunks within the same doc?

-  Should chunks be linked together with metadata to

###### Why chunking?

If your system doesn’t require
entire documents to provide
relevant answers, you can
chunk them into multiple pieces
for easier consumption (reduced
cost & latency).

Other approaches: graphs or
map-reduce

-----

## Technical patterns
##### Data preparation: embeddings

###### What to embed?

Depending on your use case
you might not want just to
embed the text in the
documents but metadata as well

-  anything that will make it easier
to surface this specific chunk or
document when performing a
search

Examples

**Embedding Q&A posts in a forum**
You might want to embed the title of the posts,
the text of the original question and the content of
the top answers.
Additionally, if the posts are tagged by topic or
with keywords, you can embed those too.

**Embedding product specs**
In additional to embedding the text contained in
documents describing the products, you might
want to add metadata that you have on the
product such as the color, size, etc. in your
embeddings.

-----

## Technical patterns
##### Data preparation: augmenting content

###### Example approaches

**Make it a guide***
Reformat the content to look more like
a step-by-step guide with clear
headings and bullet-points, as this
format is more easily understandable
by an LLM.

**Add descriptive metadata***
Consider adding keywords or text that
users might search for when thinking
of a specific product or service.

###### What does “Augmenting content” mean?

Augmenting content refers to
modifications of the original content
to make it more digestible for a
system relying on RAG. The
modifications could be a change in
format, wording, or adding
descriptive content such as
summaries or keywords

**Multimodality**
Leverage models
such as Whisper or
GPT-4V to
transform audio or
visual content into
text.
For example, you
can use GPT-4V to
generate tags for
images or to
describe slides.

-----

## Technical patterns
##### Input processing BEST PRACTICES

Consider how to transform the
input to match content in the
database

0.983, 0.123, 0.289… augment the user input

➔Comparing directly the input

red summer specificities

embeddings

Process input according to task

0.983, 0.123, 0.289…

**Q&A** 0.876, 0.145, 0.179…
HyDE: Ask LLM to hypothetically answer the
question & use the answer to search the KB

**Content search** query
Prompt LLM to rephrase input & optionally add
more context SELECT * from items…

**DB search**
NER: Find relevant entities to be used for a
keyword search or to construct a search query keywords

red summer

-----

## Technical patterns
##### Input processing: input augmentation

###### Example approaches

###### What is input augmentation?

Augmenting the input means turning
it into something different, either
rephrasing it, splitting it in several
inputs or expanding it.
This helps boost performance as
the LLM might understand better
the user intent.

**Query**
**expansion***
Rephrase the
query to be
more
descriptive

**Fallback**
Consider
implementing a
flow where the LLM
can ask for
clarification when
there is not enough
information in the
original user query
to get a result
(Especially relevant
with tool usage)

**HyDE***
Hypothetically
answer the
question & use
the answer to
search the KB

**Splitting a query in N***
When there is more than 1 question or
intent in a user query, consider
splitting it in several queries

-----

## Technical patterns
##### Input processing: NER

###### Why use NER?

Using NER (Named Entity
Recognition) allows to extract
relevant entities from the input, that
can then be used for more
deterministic search queries.
This can be useful when the scope
is very constrained.

Example

**Searching for movies**
If you have a structured database containing
metadata on movies, you can extract genre,
actors or directors names, etc. from the user
query and use this to search the database

Note: You can use exact values or embeddings after
having extracted the relevant entities

-----

## Technical patterns
##### Retrieval BEST PRACTICES

re-ranking Use a combination of semantic

where possible

+ Cache output where possible

**COMMON PITFALLS**

➔The wrong elements could be

**FINAL RESULT** compared when looking at

text similarity, that is why
re-ranking is important

re-ranking

**INPUT**

**RESULTS** **RESULTS**

Semantic

embeddings search

0.983, 0.123, 0.289…

0.876, 0.145, 0.179…

vector DB

query

SELECT * from items…

**FINAL RESULT**

keywords

relational /

red summer nosql db

-----

## Technical patterns
##### Retrieval: search

**Semantic search** **Keyword search** **Search query**

If you have extracted
specific entities or
keywords to search for,
you can search for these
in your database.

Based on the extracted
entities you have or the
user input as is, you can
construct search queries
(SQL, cypher…) and use
these queries to search
your database.

###### How to search?

**Semantic search**

Using embeddings, you

There are many different

can perform semantic

approaches to search depending on searches. You can
the use case and the existing compare embeddings

with what is in your

system. database and find the

most similar.

You can use a hybrid approach and combine several of these.
You can perform multiple searches in parallel or in sequence, or

h f k d ith th i b ddi f l

-----

## Technical patterns
##### Retrieval: multi-step retrieval

###### What is multi-step retrieval?

In some cases, there might be
several actions to be performed to
get the required information to
generate an answer.

Framework to be used:

-  When there are multiple steps to perform,
consider whether you want to handle this
yourself or use a framework to make it easier

Cost & Latency:

-  Performing multiple steps at the retrieval
stage can increase latency and cost
significantly

-  Consider performing actions in parallel to
reduce latency

Chain of Thought:

-  Guide the assistant with the chain of thought
approach: break down instructions into
several steps, with clear guidelines on
whether to continue, stop or do something
else.

-  This is more appropriate when tasks need to
b f d i ll f l “if

-----

## Technical patterns
##### Retrieval: re-ranking

###### What is re-ranking?

Re-ranking means re-ordering the
results of the retrieval process to
surface more relevant results.
This is particularly important when
doing semantic searches.

###### Example approaches

**Rule-based re-ranking**
You can use metadata to rank results by relevance. For
example, you can look at the recency of the documents, at
tags, specific keywords in the title, etc.

**Re-ranking algorithms**
There are several existing algorithms/approaches you can use
based on your use case: BERT-based re-rankers,
cross-encoder re-ranking, TF-IDF algorithms…

-----

## Technical patterns
##### Answer Generation BEST PRACTICES

Evaluate performance after each
experimentation to assess if it’s
worth exploring other paths
+ Implement guardrails if applicable

➔Going for fine-tuning without

**LLM**

**FINAL RESULT**

Prompt including

the content

Piece of content User sees the

retrieved final result

trying other approaches

➔Not paying attention to the

way the model is prompted

-----

## Technical patterns
##### Answer Generation: context window

###### How to manage context?

Depending on your use case, there are
several things to consider when
including retrieved content into the
context window to generate an answer.

Context window max size:

-  There is a maximum size, so putting too
much content is not ideal

-  In conversation use cases, the
conversation will be part of the context
as well and will add to that size

Cost & Latency vs Accuracy:

-  More context results in increased
latency and additional costs since there
will be more input tokens

-  Less context might also result in
decreased accuracy

“Lost in the middle” problem:

-  When there is too much context, LLMs
tend to forget the text “in the middle” of
the content and might look over some
important information.

-----

## Technical patterns
##### Answer Generation: optimisation


[Documentation](https://platform.openai.com/docs) [API reference](https://platform.openai.com/docs/api-reference) [Forum](https://community.openai.com/categories) Help

# Models

## Overview

The OpenAI API is powered by a diverse set of models with different capabilities and
price points. You can also make customizations to our models for your specific use
[case with fine-tuning.](https://platform.openai.com/docs/guides/fine-tuning)

MODEL DESCRIPTION

[GPT-4 and GPT-4 Turbo](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo) A set of models that improve on GPT-3.5 and can
understand as well as generate natural language or code

[GPT-3.5 Turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo) A set of models that improve on GPT-3.5 and can
understand as well as generate natural language or code

[DALL·E](https://platform.openai.com/docs/models/dall-e) A model that can generate and edit images given a natural
language prompt

[TTS](https://platform.openai.com/docs/models/tts) A set of models that can convert text into natural sounding
spoken audio

[Whisper](https://platform.openai.com/docs/models/whisper) A model that can convert audio into text

[Embeddings](https://platform.openai.com/docs/models/embeddings) A set of models that can convert text into a numerical form

[Moderation](https://platform.openai.com/docs/models/moderation) A fine-tuned model that can detect whether text may be
sensitive or unsafe

[GPT base](https://platform.openai.com/docs/models/gpt-base) A set of models without instruction following that can
understand as well as generate natural language or code

[Deprecated](https://platform.openai.com/docs/deprecations) A full list of models that have been deprecated along with
the suggested replacement

[We have also published open source models including Point-E, Whisper, Jukebox, and](https://github.com/openai/point-e)
[CLIP.](https://github.com/openai/CLIP)

## Continuous model upgrades

-----

[version. You can verify this by looking at the response object after sending a request.](https://platform.openai.com/docs/api-reference/chat/object)
The response will include the specific model version used (e.g. `gpt-3.5-turbo-`
```
0613 ).

```
We also offer static model versions that developers can continue using for at least
three months after an updated model has been introduced. With the new cadence of
model updates, we are also giving people the ability to contribute evals to help us
[improve the model for different use cases. If you are interested, check out the OpenAI](https://github.com/openai/evals)
[Evals repository.](https://github.com/openai/evals)

[Learn more about model deprecation on our deprecation page.](https://platform.openai.com/docs/deprecations)

## GPT-4 and GPT-4 Turbo

GPT-4 is a large multimodal model (accepting text or image inputs and outputting text)
that can solve difficult problems with greater accuracy than any of our previous
models, thanks to its broader general knowledge and advanced reasoning capabilities.
[GPT-4 is available in the OpenAI API to paying customers. Like](https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4) `gpt-3.5-turbo, GPT-`
[4 is optimized for chat but works well for traditional completions tasks using the Chat](https://platform.openai.com/docs/api-reference/chat)
[Completions API. Learn how to use GPT-4 in our text generation guide.](https://platform.openai.com/docs/api-reference/chat)

CONTEXT
WINDOW

128,000
tokens

128,000
tokens

128,000
tokens

TRAINING
DATA

Up to
Dec
2023

Up to
Dec
2023

Up to
Apr 2023

MODEL DESCRIPTION

`gpt-4-0125-preview` New GPT-4 Turbo

The latest GPT-4 model
intended to reduce cases of
“laziness” where the model
doesn’t complete a task.
Returns a maximum of
4,096 output tokens.
[Learn more.](https://openai.com/blog/new-embedding-models-and-api-updates)

`gpt-4-turbo-preview` Currently points to gpt-4```
                0125-preview.

```
`gpt-4-1106-preview` GPT-4 Turbo model
featuring improved
instruction following, JSON
mode, reproducible outputs,
parallel function calling, and
more. Returns a maximum
of 4,096 output tokens. This

-----

is a preview model.
[Learn more.](https://openai.com/blog/new-models-and-developer-products-announced-at-devday)

`gpt-4-vision-preview` GPT-4 with the ability to
understand images, in
addition to all other GPT-4
Turbo capabilities. Currently
points to gpt-4-1106```
                vision-preview.

```
`gpt-4-1106-vision-preview` GPT-4 with the ability to
understand images, in
addition to all other GPT-4
Turbo capabilities. Returns a
maximum of 4,096 output
tokens. This is a preview
[model version. Learn more.](https://openai.com/blog/new-models-and-developer-products-announced-at-devday)

`gpt-4` Currently points to gpt-4```
                0613. See

```
[continuous model upgrades.](https://platform.openai.com/docs/models/continuous-model-upgrades)

`gpt-4-0613` Snapshot of gpt-4 from

June 13th 2023 with
improved function calling
support.

`gpt-4-32k` Currently points to gpt-4```
                32k-0613. See

```
[continuous model upgrades.](https://platform.openai.com/docs/models/continuous-model-upgrades)
This model was never rolled
out widely in favor of GPT-4
Turbo.

`gpt-4-32k-0613` Snapshot of gpt-4-32k

from June 13th 2023 with
improved function calling
support. This model was
never rolled out widely in
favor of GPT-4 Turbo.

128,000
tokens

128,000
tokens

8,192
tokens

8,192
tokens

32,768
tokens

32,768
tokens

Up to
Apr 2023

Up to
Apr 2023

Up to
Sep 2021

Up to
Sep 2021

Up to
Sep 2021

Up to
Sep 2021

For many basic tasks, the difference between GPT-4 and GPT-3.5 models is not
significant. However, in more complex reasoning situations, GPT-4 is much more

-----

[GPT-4 outperforms both previous large language models and as of 2023, most state-](https://cdn.openai.com/papers/gpt-4.pdf)
of-the-art systems (which often have benchmark-specific training or handengineering). On the MMLU benchmark, an English-language suite of multiple-choice
questions covering 57 subjects, GPT-4 not only outperforms existing models by a
considerable margin in English, but also demonstrates strong performance in other
languages.

## GPT-3.5 Turbo

GPT-3.5 Turbo models can understand and generate natural language or code and
[have been optimized for chat using the Chat Completions API but work well for non-](https://platform.openai.com/docs/api-reference/chat)
chat tasks as well.

CONTEXT
WINDOW

16,385
tokens

4,096
tokens

16,385
tokens

TRAINING
DATA

Up to Sep
2021

Up to Sep
2021

Up to Sep
2021