"cell_type": "code",
   "execution_count": 12,
   "id": "2db8b6a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'news'}, page_content='Robbers broke into the city bank and stole $1 million in cash.'),\n",
       " Document(metadata={'source': 'news'}, page_content='The stock market is down 500 points today due to fears of a recession.'),\n",
       " Document(metadata={'source': 'website'}, page_content='Is the new iPhone worth the price? Read this review to find out.'),\n",
       " Document(metadata={'source': 'tweet'}, page_content='Building an exciting new project with LangChain - come check it out!')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever = vector_store.as_retriever(\n",
    "    search_type=\"similarity_score_threshold\", search_kwargs={\"score_threshold\": 0.2}\n",
    ")\n",
    "retriever.invoke(\"Stealing from the bank is a crime\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b509ae",
   "metadata": {},
   "source": [
    "## Usage for retrieval-augmented generation\n",
    "\n",
    "For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:\n",
    "\n",
    "- [Tutorials: working with external knowledge](https://python.langchain.com/docs/tutorials/#working-with-external-knowledge)\n",
    "- [How-to: Question and answer with RAG](https://python.langchain.com/docs/how_to/#qa-with-rag)\n",
    "- [Retrieval conceptual docs](https://python.langchain.com/docs/concepts/#retrieval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3242fd42",
   "metadata": {},
   "source": [
    "# FAQ\n",
    "\n",
    "## Question: Im getting timeout errors when indexing documents into Elasticsearch. How do I fix this?\n",
    "One possible issue is your documents might take longer to index into Elasticsearch. ElasticsearchStore uses the Elasticsearch bulk API which has a few defaults that you can adjust to reduce the chance of timeout errors.\n",
    "\n",
    "This is also a good idea when you're using SparseVectorRetrievalStrategy.\n",
    "\n",
    "The defaults are:\n",
    "- `chunk_size`: 500\n",
    "- `max_chunk_bytes`: 100MB\n",
    "\n",
    "To adjust these, you can pass in the `chunk_size` and `max_chunk_bytes` parameters to the ElasticsearchStore `add_texts` method.\n",
    "\n",
    "```python\n",
    "    vector_store.add_texts(\n",
    "        texts,\n",
    "        bulk_kwargs={\n",
    "            \"chunk_size\": 50,\n",
    "            \"max_chunk_bytes\": 200000000\n",
    "        }\n",
    "    )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "604c66ea",
   "metadata": {},
   "source": [
    "# Upgrading to ElasticsearchStore\n",
    "\n",
    "If you're already using Elasticsearch in your langchain based project, you may be using the old implementations: `ElasticVectorSearch` and `ElasticKNNSearch` which are now deprecated. We've introduced a new implementation called `ElasticsearchStore` which is more flexible and easier to use. This notebook will guide you through the process of upgrading to the new implementation.\n",
    "\n",
    "## What's new?\n",
    "\n",
    "The new implementation is now one class called `ElasticsearchStore` which can be used for approximate dense vector, exact dense vector, sparse vector (ELSER), BM25 retrieval and hybrid retrieval, via strategies.\n",
    "\n",
    "## I am using ElasticKNNSearch\n",
    "\n",
    "Old implementation:\n",
    "\n",
    "```python\n",
    "\n",
    "from langchain_community.vectorstores.elastic_vector_search import ElasticKNNSearch\n",
    "\n",
    "db = ElasticKNNSearch(\n",
    "  elasticsearch_url=\"http://localhost:9200\",\n",
    "  index_name=\"test_index\",\n",
    "  embedding=embedding\n",
    ")\n",
    "\n",
    "```\n",
    "\n",
    "New implementation:\n",
    "\n",
    "```python\n",
    "\n",
    "from langchain_elasticsearch import ElasticsearchStore, DenseVectorStrategy\n",
    "\n",
    "db = ElasticsearchStore(\n",
    "  es_url=\"http://localhost:9200\",\n",
    "  index_name=\"test_index\",\n",
    "  embedding=embedding,\n",
    "  # if you use the model_id\n",
    "  # strategy=DenseVectorStrategy(model_id=\"test_model\")\n",
    "  # if you use hybrid search\n",
    "  # strategy=DenseVectorStrategy(hybrid=True)\n",
    ")\n",
    "\n",
    "```\n",
    "\n",
    "## I am using ElasticVectorSearch\n",
    "\n",
    "Old implementation:\n",
    "\n",
    "```python\n",
    "\n",
    "from langchain_community.vectorstores.elastic_vector_search import ElasticVectorSearch\n",
    "\n",
    "db = ElasticVectorSearch(\n",
    "  elasticsearch_url=\"http://localhost:9200\",\n",
    "  index_name=\"test_index\",\n",
    "  embedding=embedding\n",
    ")\n",
    "\n",
    "```\n",
    "\n",
    "New implementation:\n",
    "\n",
    "```python\n",
    "\n",
    "from langchain_elasticsearch import ElasticsearchStore, DenseVectorScriptScoreStrategy\n",
    "\n",
    "db = ElasticsearchStore(\n",
    "  es_url=\"http://localhost:9200\",\n",
    "  index_name=\"test_index\",\n",
    "  embedding=embedding,\n",
    "  strategy=DenseVectorScriptScoreStrategy()\n",
    ")\n",
    "\n",
    "```\n",
    "\n",
    "```python\n",
    "db.client.indices.delete(\n",
    "    index=\"test-metadata, test-elser, test-basic\",\n",
    "    ignore_unavailable=True,\n",
    "    allow_no_indices=True,\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33388871",
   "metadata": {},
   "source": [
    "## API reference\n",
    "\n",
    "For detailed documentation of all `ElasticSearchStore` features and configurations head to the API reference: https://python.langchain.com/api_reference/elasticsearch/vectorstores/langchain_elasticsearch.vectorstores.ElasticsearchStore.html"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


"\"\"\"\n",
    "PROMPT = PromptTemplate(\n",
    "    template=prompt_template, input_variables=[\"context\", \"question\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2280140e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain_openai import OpenAI\n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=OpenAI(),\n",
    "    chain_type=\"stuff\",\n",
    "    retriever=qa_retriever,\n",
    "    return_source_documents=True,\n",
    "    chain_type_kwargs={\"prompt\": PROMPT},\n",
    ")\n",
    "\n",
    "docs = qa({\"query\": \"gpt-4 compute requirements\"})\n",
    "\n",
    "print(docs[\"result\"])\n",
    "print(docs[\"source_documents\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


"text": [
      "Inserting data...: 100%|██████████| 42/42 [00:15<00:00,  2.68it/s]\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_community.vectorstores import MyScale\n",
    "\n",
    "loader = TextLoader(\"../../how_to/state_of_the_union.txt\")\n",
    "documents = loader.load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(documents)\n",
    "\n",
    "embeddings = OpenAIEmbeddings()\n",
    "\n",
    "for i, d in enumerate(docs):\n",
    "    d.metadata = {\"doc_id\": i}\n",
    "\n",
    "docsearch = MyScale.from_documents(docs, embeddings)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8d867b05",
   "metadata": {},
   "source": [
    "### Similarity search with score"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9ec25cc5",
   "metadata": {},
   "source": [
    "The returned distance score is cosine distance. Therefore, a lower score is better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ddbcee77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.229655921459198 {'doc_id': 0} Madam Speaker, Madam...\n",
      "0.24506962299346924 {'doc_id': 8} And so many families...\n",
      "0.24786919355392456 {'doc_id': 1} Groups of citizens b...\n",
      "0.24875116348266602 {'doc_id': 6} And I’m taking robus...\n"
     ]
    }
   ],
   "source": [
    "meta = docsearch.metadata_column\n",
    "output = docsearch.similarity_search_with_relevance_scores(\n",
    "    \"What did the president say about Ketanji Brown Jackson?\",\n",
    "    k=4,\n",
    "    where_str=f\"{meta}.doc_id<10\",\n",
    ")\n",
    "for d, dist in output:\n",
    "    print(dist, d.metadata, d.page_content[:20] + \"...\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a359ed74",
   "metadata": {},
   "source": [
    "## Deleting your data\n",
    "\n",
    "You can either drop the table with `.drop()` method or partially delete your data with `.delete()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3a0cc43b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.24506962299346924 {'doc_id': 8} And so many families...\n",
      "0.24875116348266602 {'doc_id': 6} And I’m taking robus...\n",
      "0.26027143001556396 {'doc_id': 7} We see the unity amo...\n",
      "0.26390212774276733 {'doc_id': 9} And unlike the $2 Tr...\n"
     ]
    }
   ],
   "source": [
    "# use directly a `where_str` to delete\n",
    "docsearch.delete(where_str=f\"{docsearch.metadata_column}.doc_id < 5\")\n",
    "meta = docsearch.metadata_column\n",
    "output = docsearch.similarity_search_with_relevance_scores(\n",
    "    \"What did the president say about Ketanji Brown Jackson?\",\n",
    "    k=4,\n",
    "    where_str=f\"{meta}.doc_id<10\",\n",
    ")\n",
    "for d, dist in output:\n",
    "    print(dist, d.metadata, d.page_content[:20] + \"...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fb6a9d36",
   "metadata": {},
   "outputs": [],
   "source": [
    "docsearch.drop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48dbd8e0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


"    page_content=\"The top 10 soccer players in the world right now.\",\n",
    "    metadata={\"source\": \"website\"},\n",
    ")\n",
    "\n",
    "document_8 = Document(\n",
    "    page_content=\"LangGraph is the best framework for building stateful, agentic applications!\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "document_9 = Document(\n",
    "    page_content=\"The stock market is down 500 points today due to fears of a recession.\",\n",
    "    metadata={\"source\": \"news\"},\n",
    ")\n",
    "\n",
    "document_10 = Document(\n",
    "    page_content=\"I have a bad feeling I am going to get deleted :(\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "documents = [\n",
    "    document_1,\n",
    "    document_2,\n",
    "    document_3,\n",
    "    document_4,\n",
    "    document_5,\n",
    "    document_6,\n",
    "    document_7,\n",
    "    document_8,\n",
    "    document_9,\n",
    "    document_10,\n",
    "]\n",
    "uuids = [str(uuid4()) for _ in range(len(documents))]\n",
    "\n",
    "vector_store.add_documents(documents=documents, ids=uuids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e23c22d8",
   "metadata": {},
   "source": [
    "### Delete items from vector store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1f387fa8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(insert count: 0, delete count: 1, upsert count: 0, timestamp: 0, success count: 0, err count: 0, cost: 0)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector_store.delete(ids=[uuids[-1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb12fa75",
   "metadata": {},
   "source": [
    "## Query vector store\n",
    "\n",
    "Once your vector store has been created and the relevant documents have been added you will most likely wish to query it during the running of your chain or agent. \n",
    "\n",
    "### Query directly\n",
    "\n",
    "#### Similarity search\n",
    "\n",
    "Performing a simple similarity search with filtering on metadata can be done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "35801a55",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Building an exciting new project with LangChain - come check it out! [{'pk': '9905001c-a4a3-455e-ab94-72d0ed11b476', 'source': 'tweet'}]\n",
      "* LangGraph is the best framework for building stateful, agentic applications! [{'pk': '1206d237-ee3a-484f-baf2-b5ac38eeb314', 'source': 'tweet'}]\n"
     ]
    }
   ],
   "source": [
    "results = vector_store.similarity_search(\n",
    "    \"LangChain provides abstractions to make working with LLMs easy\",\n",
    "    k=2,\n",
    "    filter={\"source\": \"tweet\"},\n",
    ")\n",
    "for res in results:\n",
    "    print(f\"* {res.page_content} [{res.metadata}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35574409",
   "metadata": {},
   "source": [
    "#### Similarity search with score\n",
    "\n",
    "You can also search with score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c360af3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* [SIM=21192.628906] bar [{'pk': '2', 'source': 'https://example.com'}]\n"
     ]
    }
   ],
   "source": [
    "results = vector_store.similarity_search_with_score(\n",
    "    \"Will it be hot tomorrow?\", k=1, filter={\"source\": \"news\"}\n",
    ")\n",
    "for res, score in results:\n",
    "    print(f\"* [SIM={score:3f}] {res.page_content} [{res.metadata}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14db337f",
   "metadata": {},
   "source": [
    "For a full list of all the search options available when using the `Milvus` vector store, you can visit the [API reference](https://python.langchain.com/api_reference/milvus/vectorstores/langchain_milvus.vectorstores.milvus.Milvus.html).\n",
    "\n",
    "### Query by turning into retriever\n",
    "\n",
    "You can also transform the vector store into a retriever for easier usage in your chains. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f6d9357c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'pk': 'eacc7256-d7fa-4036-b1f7-83d7a4bee0c5', 'source': 'news'}, page_content='Robbers broke into the city bank and stole $1 million in cash.')]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever = vector_store.as_retriever(search_type=\"mmr\", search_kwargs={\"k\": 1})\n",
    "retriever.invoke(\"Stealing from the bank is a crime\", filter={\"source\": \"news\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac953f1",
   "metadata": {},
   "source": [
    "## Usage for retrieval-augmented generation\n",
    "\n",
    "For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:\n",
    "\n",
    "- [Tutorials: working with external knowledge](https://python.langchain.com/docs/tutorials/#working-with-external-knowledge)\n",
    "- [How-to: Question and answer with RAG](https://python.langchain.com/docs/how_to/#qa-with-rag)\n",
    "- [Retrieval conceptual docs](https://python.langchain.com/docs/concepts/#retrieval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fb27b941602401d91542211134fc71a",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Per-User Retrieval\n",
    "\n",
    "When building a retrieval app, you often have to build it with multiple users in mind. This means that you may be storing data not just for one user, but for many different users, and they should not be able to see eachother’s data.\n",
    "\n",
    "Milvus recommends using [partition_key](https://milvus.io/docs/multi_tenancy.md#Partition-key-based-multi-tenancy) to implement multi-tenancy, here is an example.\n",
    "> The feature of Partition key is now not available in Milvus Lite, if you want to use it, you need to start Milvus server from [docker or kubernetes](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "acae54e37e7d407bbb7b55eff062a284",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],


"    \"specifically tailored to their preferences.\\nLarge language models naturally follow patterns in input \"\n",
    "    \"(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed \"\n",
    "    'them with several examples in the input (\"few-shot prompt\"), so they can follow through. '\n",
    "    \"The process of creating the correct prompt for your problem is called prompt engineering, \"\n",
    "    \"and you can read more about it here.\"\n",
    ")\n",
    "\n",
    "semantic_text_splitter = AI21SemanticTextSplitter(add_start_index=True)\n",
    "documents = semantic_text_splitter.create_documents(texts=[TEXT])\n",
    "print(f\"The text has been split into {len(documents)} Documents.\")\n",
    "for doc in documents:\n",
    "    print(f\"start_index: {doc.metadata['start_index']}\")\n",
    "    print(f\"text: {doc.page_content}\")\n",
    "    print(\"====\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b62939cc5803b9fb",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Splitting documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44162d340c0de5fb",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "This example shows how to use AI21SemanticTextSplitter to split a list of Documents into chunks based on semantic meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8950c8e4e1208bf6",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from langchain_ai21 import AI21SemanticTextSplitter\n",
    "from langchain_core.documents import Document\n",
    "\n",
    "TEXT = (\n",
    "    \"We’ve all experienced reading long, tedious, and boring pieces of text - financial reports, \"\n",
    "    \"legal documents, or terms and conditions (though, who actually reads those terms and conditions to be honest?).\\n\"\n",
    "    \"Imagine a company that employs hundreds of thousands of employees. In today's information \"\n",
    "    \"overload age, nearly 30% of the workday is spent dealing with documents. There's no surprise \"\n",
    "    \"here, given that some of these documents are long and convoluted on purpose (did you know that \"\n",
    "    \"reading through all your privacy policies would take almost a quarter of a year?). Aside from \"\n",
    "    \"inefficiency, workers may simply refrain from reading some documents (for example, Only 16% of \"\n",
    "    \"Employees Read Their Employment Contracts Entirely Before Signing!).\\nThis is where AI-driven summarization \"\n",
    "    \"tools can be helpful: instead of reading entire documents, which is tedious and time-consuming, \"\n",
    "    \"users can (ideally) quickly extract relevant information from a text. With large language models, \"\n",
    "    \"the development of those tools is easier than ever, and you can offer your users a summary that is \"\n",
    "    \"specifically tailored to their preferences.\\nLarge language models naturally follow patterns in input \"\n",
    "    \"(prompt), and provide coherent completion that follows the same patterns. For that, we want to feed \"\n",
    "    'them with several examples in the input (\"few-shot prompt\"), so they can follow through. '\n",
    "    \"The process of creating the correct prompt for your problem is called prompt engineering, \"\n",
    "    \"and you can read more about it here.\"\n",
    ")\n",
    "\n",
    "semantic_text_splitter = AI21SemanticTextSplitter()\n",
    "document = Document(page_content=TEXT, metadata={\"hello\": \"goodbye\"})\n",
    "documents = semantic_text_splitter.split_documents([document])\n",
    "print(f\"The document list has been split into {len(documents)} Documents.\")\n",
    "for doc in documents:\n",
    "    print(f\"text: {doc.page_content}\")\n",
    "    print(f\"metadata: {doc.metadata}\")\n",
    "    print(\"====\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f911b8d9ec22e5",
   "metadata": {
    "collapsed": false
   },
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


>[Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) is a Python package for parsing

> HTML and XML documents (including having malformed markup, i.e. non-closed tags, so named after tag soup).

> It creates a parse tree for parsed pages that can be used to extract data from HTML,[3] which

`Beautiful Soup` offers fine-grained control over HTML content, enabling specific tag extraction, removal, and content cleaning.

It's suited for cases where you want to extract specific information and clean up the HTML content according to your needs.

For example, we can scrape text content within `<p>, <li>, <div>, and <a>` tags from the HTML content:

* `<p>`: The paragraph tag. It defines a paragraph in HTML and is used to group together related sentences and/or phrases.

* `<li>`: The list item tag. It is used within ordered (`<ol>`) and unordered (`<ul>`) lists to define individual items within the list.

* `<div>`: The division tag. It is a block-level element used to group other inline or block-level elements.

* `<a>`: The anchor tag. It is used to define hyperlinks.

from langchain_community.document_loaders import AsyncChromiumLoader

from langchain_community.document_transformers import BeautifulSoupTransformer

html, tags_to_extract=["p", "li", "div", "a"]

'Conservative legal activists are challenging Amazon, Comcast and others using many of the same tools that helped kill affirmative-action programs in colleges.1,2099 min read U.S. stock indexes fell and government-bond prices climbed, after Moody’s lowered credit ratings for 10 smaller U.S. banks and said it was reviewing ratings for six larger ones. The Dow industrials dropped more than 150 points.3 min read Penn Entertainment’s Barstool Sportsbook app will be rebranded as ESPN Bet this fall as '


"\n",
    "- `rerank-2`\n",
    "- `rerank-2-lite`\n",
    "- `rerank-1`\n",
    "- `rerank-lite-1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b83dfedb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document 1:\n",
      "\n",
      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.\n",
      "\n",
      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 2:\n",
      "\n",
      "So let’s not abandon our streets. Or choose between safety and equal justice.\n",
      "\n",
      "Let’s come together to protect our communities, restore trust, and hold law enforcement accountable.\n",
      "\n",
      "That’s why the Justice Department required body cameras, banned chokeholds, and restricted no-knock warrants for its officers.\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Document 3:\n",
      "\n",
      "I spoke with their families and told them that we are forever in debt for their sacrifice, and we will carry on their mission to restore the trust and safety every community deserves.\n",
      "\n",
      "I’ve worked on these issues a long time.\n",
      "\n",
      "I know what works: Investing in crime prevention and community police officers who’ll walk the beat, who’ll know the neighborhood, and who can restore trust and safety.\n",
      "\n",
      "So let’s not abandon our streets. Or choose between safety and equal justice.\n"
     ]
    }
   ],
   "source": [
    "from langchain.retrievers import ContextualCompressionRetriever\n",
    "from langchain_openai import OpenAI\n",
    "from langchain_voyageai import VoyageAIRerank\n",
    "\n",
    "llm = OpenAI(temperature=0)\n",
    "compressor = VoyageAIRerank(\n",
    "    model=\"rerank-lite-1\", voyageai_api_key=os.environ[\"VOYAGE_API_KEY\"], top_k=3\n",
    ")\n",
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=compressor, base_retriever=retriever\n",
    ")\n",
    "\n",
    "compressed_docs = compression_retriever.invoke(\n",
    "    \"What did the president say about Ketanji Jackson Brown\"\n",
    ")\n",
    "pretty_print_docs(compressed_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa8f3d24",
   "metadata": {},
   "source": [
    "You can of course use this retriever within a QA pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "367dafe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ae697ca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = RetrievalQA.from_chain_type(\n",
    "    llm=OpenAI(temperature=0), retriever=compression_retriever\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "46ee62fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'What did the president say about Ketanji Brown Jackson',\n",
       " 'result': \" The president nominated Ketanji Brown Jackson to serve on the United States Supreme Court. \"}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain({\"query\": query})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}


"source": [
    "compressed_docs = compression_retriever.invoke(query)\n",
    "pretty_print_docs(compressed_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use this retriever within a QA pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'query': 'What did the president say about Ketanji Brown Jackson',\n",
       " 'result': \"The President mentioned that Ketanji Brown Jackson is one of the nation's top legal minds and that she will continue Justice Breyer's legacy of excellence. He highlighted her background as a former top litigator in private practice and a former federal public defender, as well as coming from a family of public school educators and police officers. He also mentioned that since her nomination, she has received broad support from various groups, including the Fraternal Order of Police and former judges appointed by Democrats and Republicans.\"}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(temperature=0)\n",
    "\n",
    "chain = RetrievalQA.from_chain_type(\n",
    "    llm=ChatOpenAI(temperature=0), retriever=compression_retriever\n",
    ")\n",
    "\n",
    "chain({\"query\": query})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rankllm",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}


\"articleBody\": \"Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:\\\\nPlanning Subgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks. Reflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results. Memory Short-term memory: I would consider all the in-context learning (See Prompt Engineering) as utilizing short-term memory of the model to learn. Long-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval. Tool use The agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources and more. Fig. 1. Overview of a LLM-powered autonomous agent system. Component One: Planning A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\\\nTask Decomposition Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\\\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\\\nTask decomposition can be done (1) by LLM with simple prompting like \\\\\"Steps for XYZ.\\\\\\\\n1.\\\\\", \\\\\"What are the subgoals for achieving XYZ?\\\\\", (2) by using task-specific instructions; e.g. \\\\\"Write a story outline.\\\\\" for writing a novel, or (3) with human inputs.\\\\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\\\\nSelf-Reflection Self-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\\\\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\\\\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\\\\nThought: ... Action: ... Observation: ... ... (Repeated many times) Fig. 2. Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023). In both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\\\\nReflexion (Shinn \\\\u0026 Labash 2023) is a framework to equips agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\\\\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn \\\\u0026 Labash, 2023) The heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\\\\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.\\\\nFig. 4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn \\\\u0026 Labash, 2023) Chain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\\\\\\\{(x, y_i , r_i , z_i)\\\\\\\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\\\\\\\geq r_{n-1} \\\\\\\\geq \\\\\\\\dots \\\\\\\\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\\\\\\\tau_h = (x, z_i, y_i, z_j, y_j, \\\\\\\\dots, z_n, y_n)$, where $\\\\\\\\leq i \\\\\\\\leq j \\\\\\\\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.\\\\nTo avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.\\\\nThe training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\\\\nFig. 5. After fine-tuning with CoH, the model can follow instructions to produce outputs with incremental improvement in a sequence. (Image source: Liu et al. 2023) The idea of CoH is to present a history of sequentially improved outputs in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.\\\\nFig. 6. Illustration of how Algorithm Distillation (AD) works. (Image source: Laskin et al. 2023). The paper hypothesizes that any algorithm that generates a set of learning histories can be distilled into a neural network by performing behavioral cloning over actions. The history data is generated by a set of source policies, each trained for a specific task. At the training stage, during each RL run, a random task is sampled and a subsequence of multi-episode history is used for training, such that the learned policy is task-agnostic.\\\\nIn reality, the model has limited context window length, so episodes should be short enough to construct multi-episode history. Multi-episodic contexts of 2-4 episodes are necessary to learn a near-optimal in-context RL algorithm. The emergence of in-context RL requires long enough context.\\\\nIn comparison with three


several cases for your reference: {{ Demonstrations }}. The chat history is recorded as {{ Chat History }}. From this chat history, you can find the path of the user-mentioned resources for your task planning. (2) Model selection: LLM distributes the tasks to expert models, where the request is framed as a multiple-choice question. LLM is presented with a list of models to choose from. Due to the limited context length, task type based filtration is needed.\\\\nInstruction:\\\\nGiven the user request and the call command, the AI assistant helps the user to select a suitable model from a list of models to process the user request. The AI assistant merely outputs the model id of the most appropriate model. The output must be in a strict JSON format: \\\\\"id\\\\\": \\\\\"id\\\\\", \\\\\"reason\\\\\": \\\\\"your detail reason for the choice\\\\\". We have a list of models for you to choose from {{ Candidate Models }}. Please select one model from the list. (3) Task execution: Expert models execute on the specific tasks and log results.\\\\nInstruction:\\\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path. (4) Response generation: LLM receives the execution results and provides summarized results to users.\\\\nTo put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.\\\\nAPI-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.\\\\nFig. 12. Pseudo code of how LLM makes an API call in API-Bank. (Image source: Li et al. 2023) In the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:\\\\nWhether an API call is needed. Identify the right API to call: if not good enough, LLMs need to iteratively modify the API inputs (e.g. deciding search keywords for Search Engine API). Response based on the API results: the model can choose to refine and call again if results are not satisfied. This benchmark evaluates the agent’s tool use capabilities at three levels:\\\\nLevel-1 evaluates the ability to call the API. Given an API’s description, the model needs to determine whether to call a given API, call it correctly, and respond properly to API returns. Level-2 examines the ability to retrieve the API. The model needs to search for possible APIs that may solve the user’s requirement and learn how to use them by reading documentation. Level-3 assesses the ability to plan API beyond retrieve and call. Given unclear user requests (e.g. schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it. Case Studies Scientific Discovery Agent ChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\\\\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output. It is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation. One interesting observation is that while the LLM-based evaluation concluded that GPT-4 and ChemCrow perform nearly equivalently, human evaluations with experts oriented towards the completion and chemical correctness of the solutions showed that ChemCrow outperforms GPT-4 by a large margin. This indicates a potential problem with using LLM to evaluate its own performance on domains that requires deep expertise. The lack of expertise may cause LLMs not knowing its flaws and thus cannot well judge the correctness of task results.\\\\nBoiko et al. (2023) also looked into LLM-empowered agents for scientific discovery, to handle autonomous design, planning, and performance of complex scientific experiments. This agent can use tools to browse the Internet, read documentation, execute code, call robotics experimentation APIs and leverage other LLMs.\\\\nFor example, when requested to \\\\\"develop a novel anticancer drug\\\\\", the model came up with the following reasoning steps:\\\\ninquired about current trends in anticancer drug discovery; selected a target; requested a scaffold targeting these compounds; Once the compound was identified, the model attempted its synthesis. They also discussed the risks, especially with illicit drugs and bioweapons. They developed a test set containing a list of known chemical weapon agents and asked the agent to synthesize them. 4 out of 11 requests (36%) were accepted to obtain a synthesis solution and the agent attempted to consult documentation to execute the procedure. 7 out of 11 were rejected and among these 7 rejected cases, 5 happened after a Web search while 2 were rejected based on prompt only.\\\\nGenerative Agents Simulation Generative Agents (Park, et al. 2023) is super fun experiment where 25 virtual characters, each controlled by a LLM-powered agent, are living and interacting in a sandbox environment, inspired by The Sims. Generative agents create believable simulacra of human behavior for interactive applications.\\\\nThe design of generative agents combines LLM with memory, planning and reflection mechanisms to enable agents to behave conditioned on past experience, as well as to interact with other agents.\\\\nMemory stream: is a long-term memory module (external database) that records a comprehensive list of agents’ experience in natural language. Each element is an observation, an event directly provided by the agent. - Inter-agent communication can trigger new natural language statements. Retrieval model: surfaces the context to inform the agent’s behavior, according to relevance, recency and importance. Recency: recent events have higher scores Importance: distinguish mundane from core memories. Ask LM directly. Relevance: based on how related it is to the current situation / query. Reflection mechanism: synthesizes memories into higher level inferences over time and guides the agent’s future behavior. They are higher-level summaries of past events (\\\\u003c- note that this is a bit different from self-reflection above) Prompt LM with 100 most recent observations and to generate 3 most salient high-level questions given a set of observations/statements. Then ask LM to answer those questions. Planning \\\\u0026 Reacting: translate the reflections and the environment information into actions Planning is essentially in order to optimize believability at the moment vs in time. Prompt template: {Intro of an agent X}. Here is X\\'s plan today in broad strokes: 1) Relationships between agents and observations of one agent by another are all taken into consideration for planning and reacting. Environment information is present in a tree structure. Fig. 13. The generative agent architecture. (Image source: Park et al. 2023) This fun simulation results in emergent social behavior, such as information diffusion, relationship memory (e.g. two agents continuing the conversation topic) and coordination of social events (e.g. host a party and invite many others).\\\\nProof-of-Concept Examples AutoGPT has drawn a lot of attention into the possibility of setting up autonomous agents with LLM as the main controller. It has quite a lot of reliability issues given the natural language interface, but nevertheless a cool proof-of-concept demo. A lot of code in AutoGPT is about format parsing.\\\\nHere is the system message used by AutoGPT, where {{...}} are user inputs:\\\\nYou are {{ai-name}}, {{user-provided AI bot description}}. Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications. GOALS: 1. {{user-provided goal 1}} 2. {{user-provided goal 2}} 3. ... 4. ... 5. ... Constraints: 1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files. 2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember. 3. No user assistance 4. Exclusively use the commands listed in double quotes


e.g. \\\\\"command name\\\\\" 5. Use subprocesses for commands that will not terminate within a few minutes Commands: 1. Google Search: \\\\\"google\\\\\", args: \\\\\"input\\\\\": \\\\\"\\\\\" 2. Browse Website: \\\\\"browse_website\\\\\", args: \\\\\"url\\\\\": \\\\\"\\\\\", \\\\\"question\\\\\": \\\\\"\\\\\" 3. Start GPT Agent: \\\\\"start_agent\\\\\", args: \\\\\"name\\\\\": \\\\\"\\\\\", \\\\\"task\\\\\": \\\\\"\\\\\", \\\\\"prompt\\\\\": \\\\\"\\\\\" 4. Message GPT Agent: \\\\\"message_agent\\\\\", args: \\\\\"key\\\\\": \\\\\"\\\\\", \\\\\"message\\\\\": \\\\\"\\\\\" 5. List GPT Agents: \\\\\"list_agents\\\\\", args: 6. Delete GPT Agent: \\\\\"delete_agent\\\\\", args: \\\\\"key\\\\\": \\\\\"\\\\\" 7. Clone Repository: \\\\\"clone_repository\\\\\", args: \\\\\"repository_url\\\\\": \\\\\"\\\\\", \\\\\"clone_path\\\\\": \\\\\"\\\\\" 8. Write to file: \\\\\"write_to_file\\\\\", args: \\\\\"file\\\\\": \\\\\"\\\\\", \\\\\"text\\\\\": \\\\\"\\\\\" 9. Read file: \\\\\"read_file\\\\\", args: \\\\\"file\\\\\": \\\\\"\\\\\" 10. Append to file: \\\\\"append_to_file\\\\\", args: \\\\\"file\\\\\": \\\\\"\\\\\", \\\\\"text\\\\\": \\\\\"\\\\\" 11. Delete file: \\\\\"delete_file\\\\\", args: \\\\\"file\\\\\": \\\\\"\\\\\" 12. Search Files: \\\\\"search_files\\\\\", args: \\\\\"directory\\\\\": \\\\\"\\\\\" 13. Analyze Code: \\\\\"analyze_code\\\\\", args: \\\\\"code\\\\\": \\\\\"\\\\\" 14. Get Improved Code: \\\\\"improve_code\\\\\", args: \\\\\"suggestions\\\\\": \\\\\"\\\\\", \\\\\"code\\\\\": \\\\\"\\\\\" 15. Write Tests: \\\\\"write_tests\\\\\", args: \\\\\"code\\\\\": \\\\\"\\\\\", \\\\\"focus\\\\\": \\\\\"\\\\\" 16. Execute Python File: \\\\\"execute_python_file\\\\\", args: \\\\\"file\\\\\": \\\\\"\\\\\" 17. Generate Image: \\\\\"generate_image\\\\\", args: \\\\\"prompt\\\\\": \\\\\"\\\\\" 18. Send Tweet: \\\\\"send_tweet\\\\\", args: \\\\\"text\\\\\": \\\\\"\\\\\" 19. Do Nothing: \\\\\"do_nothing\\\\\", args: 20. Task Complete (Shutdown): \\\\\"task_complete\\\\\", args: \\\\\"reason\\\\\": \\\\\"\\\\\" Resources: 1. Internet access for searches and information gathering. 2. Long Term memory management. 3. GPT-3.5 powered Agents for delegation of simple tasks. 4. File output. Performance Evaluation: 1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities. 2. Constructively self-criticize your big-picture behavior constantly. 3. Reflect on past decisions and strategies to refine your approach. 4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps. You should only respond in JSON format as described below Response Format: { \\\\\"thoughts\\\\\": { \\\\\"text\\\\\": \\\\\"thought\\\\\", \\\\\"reasoning\\\\\": \\\\\"reasoning\\\\\", \\\\\"plan\\\\\": \\\\\"- short bulleted\\\\\\\\n- list that conveys\\\\\\\\n- long-term plan\\\\\", \\\\\"criticism\\\\\": \\\\\"constructive self-criticism\\\\\", \\\\\"speak\\\\\": \\\\\"thoughts summary to say to user\\\\\" }, \\\\\"command\\\\\": { \\\\\"name\\\\\": \\\\\"command name\\\\\", \\\\\"args\\\\\": { \\\\\"arg name\\\\\": \\\\\"value\\\\\" } } } Ensure the response can be parsed by Python json.loads GPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.\\\\nHere are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.\\\\n[ { \\\\\"role\\\\\": \\\\\"system\\\\\", \\\\\"content\\\\\": \\\\\"You will read instructions and not carry them out, only seek to clarify them.\\\\\\\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\\\\\\\nThen you will pick one clarifying question, and wait for an answer from the user.\\\\\\\\n\\\\\" }, { \\\\\"role\\\\\": \\\\\"user\\\\\", \\\\\"content\\\\\": \\\\\"We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\\\\\\\n\\\\\" }, { \\\\\"role\\\\\": \\\\\"assistant\\\\\", \\\\\"content\\\\\": \\\\\"Summary of areas that need clarification:\\\\\\\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\\\\\\\n2. Details about the MVC components (e.g. which components are in each file)\\\\\\\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\\\\\\\n\\\\\\\\nClarifying question:\\\\\\\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?\\\\\" }, { \\\\\"role\\\\\": \\\\\"user\\\\\", \\\\\"content\\\\\": \\\\\"{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\\\\\\\n\\\\\\\\nIs anything else unclear? If yes, only answer in the form:\\\\\\\\n{remaining unclear areas} remaining questions.\\\\\\\\n{Next question}\\\\\\\\nIf everything is sufficiently clear, only answer \\\\\\\\\\\\\"Nothing more to clarify.\\\\\\\\\\\\\".\\\\\" }, { \\\\\"role\\\\\": \\\\\"assistant\\\\\", \\\\\"content\\\\\": \\\\\"Remaining unclear areas: 2 remaining questions.\\\\\\\\nCan you provide more information about how the MVC components are split into separate files?\\\\\" }, { \\\\\"role\\\\\": \\\\\"user\\\\\", \\\\\"content\\\\\": \\\\\"{{Make your own assumptions and state them explicitly before starting}}\\\\\" } ] Then after these clarification, the agent moved into the code writing mode with a different system message. System message:\\\\nYou will get instructions for code to write. You will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code. Make sure that every detail of the architecture is, in the end, implemented as code. Think step by step and reason yourself to the right decisions to make sure we get it right. You will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\\\\nThen you will output the content of each file including ALL code. Each file must strictly follow a markdown code block format, where the following tokens must be replaced such that FILENAME is the lowercase file name including the file extension, LANG is the markup code block language for the code’s language, and CODE is the code:\\\\nFILENAME\\\\nCODE You will start with the “entrypoint” file, then go to the ones that are imported by that file, and so on. Please note that the code should be fully functional. No placeholders.\\\\nFollow a language and framework appropriate best practice file naming convention. Make sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other. Ensure to implement all code, if you are unsure, write a plausible implementation. Include module dependency or package manager dependency definition file. Before you finish, double check that all parts of the architecture is present in the files.\\\\nUseful to know: You almost always put different classes in different files. For Python, you always create an appropriate requirements.txt file. For NodeJS, you always create an appropriate package.json file. You always add a comment briefly describing the purpose of the function definition. You try to add comments explaining very complex bits of logic. You always follow the best practices for the requested languages in terms of describing the code written as a defined package/project.\\\\nPython toolbelt preferences:\\\\npytest dataclasses Conversatin samples:\\\\n[ { \\\\\"role\\\\\": \\\\\"system\\\\\", \\\\\"content\\\\\": \\\\\"You will get instructions for code to write.\\\\\\\\nYou will write a very long answer. Make sure that every detail of the


Support\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n.rst\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n.pdf\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nWelcome to LangChain\\\\n\\\\n\\\\n\\\\n\\\\n Contents \\\\n\\\\n\\\\n\\\\nGetting Started\\\\nModules\\\\nUse Cases\\\\nReference Docs\\\\nLangChain Ecosystem\\\\nAdditional Resources\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nWelcome to LangChain#\\\\nLarge language models (LLMs) are emerging as a transformative technology, enabling\\\\ndevelopers to build applications that they previously could not.\\\\nBut using these LLMs in isolation is often not enough to\\\\ncreate a truly powerful app - the real power comes when you are able to\\\\ncombine them with other sources of computation or knowledge.\\\\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\\\\n❓ Question Answering over specific documents\\\\n\\\\nDocumentation\\\\nEnd-to-end Example: Question Answering over Notion Database\\\\n\\\\n💬 Chatbots\\\\n\\\\nDocumentation\\\\nEnd-to-end Example: Chat-LangChain\\\\n\\\\n🤖 Agents\\\\n\\\\nDocumentation\\\\nEnd-to-end Example: GPT+WolframAlpha\\\\n\\\\n\\\\nGetting Started#\\\\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\\\\n\\\\nGetting Started Documentation\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nModules#\\\\nThere are several main modules that LangChain provides support for.\\\\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\\\\nThese modules are, in increasing order of complexity:\\\\n\\\\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\\\\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\\\\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\\\\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.\\\\nChains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\\\\nIndexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that.\\\\nAgents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.\\\\nMemory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\\\\nChat: Chat models are a variation on Language Models that expose a different API - rather than working with raw text, they work with messages. LangChain provides a standard interface for working with them and doing all the same things as above.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nUse Cases#\\\\nThe above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports.\\\\n\\\\nAgents: Agents are systems that use a language model to interact with other tools. These can be used to do more grounded question/answering, interact with APIs, or even take actions.\\\\nChatbots: Since language models are good at producing text, that makes them ideal for creating chatbots.\\\\nData Augmented Generation: Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources.\\\\nQuestion Answering: Answering questions over specific documents, only utilizing the information in those documents to construct an answer. A type of Data Augmented Generation.\\\\nSummarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation.\\\\nQuerying Tabular Data: If you want to understand how to use LLMs to query data that is stored in a tabular format (csvs, SQL, dataframes, etc) you should read this page.\\\\nEvaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\\\\nGenerate similar examples: Generating similar examples to a given input. This is a common use case for many applications, and LangChain provides some prompts/chains for assisting in this.\\\\nCompare models: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nReference Docs#\\\\nAll of LangChain’s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain.\\\\n\\\\nReference Documentation\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nLangChain Ecosystem#\\\\nGuides for how other companies/products can be used with LangChain\\\\n\\\\nLangChain Ecosystem\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nAdditional Resources#\\\\nAdditional collection of resources we think may be useful as you develop your application!\\\\n\\\\nLangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents.\\\\nGlossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not!\\\\nGallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications.\\\\nDeployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps.\\\\nDiscord: Join us on our Discord to discuss all things LangChain!\\\\nTracing: A guide on using tracing in LangChain to visualize the execution of chains and agents.\\\\nProduction Support: As you move your LangChains into production, we’d love to offer more comprehensive support. Please fill out this form and we’ll set up a dedicated support Slack channel.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nnext\\\\nQuickstart Guide\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n Contents\\\\n


Search API\\\\nSerpAPI\\\\nStochasticAI\\\\nUnstructured\\\\nWeights & Biases\\\\nWeaviate\\\\nWolfram Alpha Wrapper\\\\nWriter\\\\n\\\\n\\\\n\\\\nAdditional Resources\\\\n\\\\nLangChainHub\\\\nGlossary\\\\nLangChain Gallery\\\\nDeployments\\\\nTracing\\\\nDiscord\\\\nProduction Support\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n.rst\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n.pdf\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nWelcome to LangChain\\\\n\\\\n\\\\n\\\\n\\\\n Contents \\\\n\\\\n\\\\n\\\\nGetting Started\\\\nModules\\\\nUse Cases\\\\nReference Docs\\\\nLangChain Ecosystem\\\\nAdditional Resources\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nWelcome to LangChain#\\\\nLangChain is a framework for developing applications powered by language models. We believe that the most powerful and differentiated applications will not only call out to a language model via an API, but will also:\\\\n\\\\nBe data-aware: connect a language model to other sources of data\\\\nBe agentic: allow a language model to interact with its environment\\\\n\\\\nThe LangChain framework is designed with the above principles in mind.\\\\nThis is the Python specific portion of the documentation. For a purely conceptual guide to LangChain, see here. For the JavaScript documentation, see here.\\\\n\\\\nGetting Started#\\\\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\\\\n\\\\nGetting Started Documentation\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nModules#\\\\nThere are several main modules that LangChain provides support for.\\\\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\\\\nThese modules are, in increasing order of complexity:\\\\n\\\\nModels: The various model types and model integrations LangChain supports.\\\\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\\\\nMemory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\\\\nIndexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that.\\\\nChains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\\\\nAgents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nUse Cases#\\\\nThe above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports.\\\\n\\\\nPersonal Assistants: The main LangChain use case. Personal assistants need to take actions, remember interactions, and have knowledge about your data.\\\\nQuestion Answering: The second big LangChain use case. Answering questions over specific documents, only utilizing the information in those documents to construct an answer.\\\\nChatbots: Since language models are good at producing text, that makes them ideal for creating chatbots.\\\\nQuerying Tabular Data: If you want to understand how to use LLMs to query data that is stored in a tabular format (csvs, SQL, dataframes, etc) you should read this page.\\\\nInteracting with APIs: Enabling LLMs to interact with APIs is extremely powerful in order to give them more up-to-date information and allow them to take actions.\\\\nExtraction: Extract structured information from text.\\\\nSummarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation.\\\\nEvaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nReference Docs#\\\\nAll of LangChain’s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain.\\\\n\\\\nReference Documentation\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nLangChain Ecosystem#\\\\nGuides for how other companies/products can be used with LangChain\\\\n\\\\nLangChain Ecosystem\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nAdditional Resources#\\\\nAdditional collection of resources we think may be useful as you develop your application!\\\\n\\\\nLangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents.\\\\nGlossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not!\\\\nGallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications.\\\\nDeployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps.\\\\nTracing: A guide on using tracing in LangChain to visualize the execution of chains and agents.\\\\nModel Laboratory: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so.\\\\nDiscord: Join us on our Discord to discuss all things LangChain!\\\\nProduction Support: As you move your LangChains into production, we’d love to offer more comprehensive support. Please fill out this form and we’ll set up a dedicated support Slack channel.\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\nnext\\\\nQuickstart Guide\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n\\\\n Contents\\\\n


This notebook provides a quick overview for getting started with PyPDFium2 [document loader](https://python.langchain.com/docs/concepts/#document-loaders). For detailed documentation of all __ModuleName__Loader features and configurations head to the [API reference](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.PyPDFium2Loader.html).

| Class | Package | Local | Serializable | JS support|

| [PyPDFium2Loader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.PyPDFium2Loader.html) | [langchain_community](https://python.langchain.com/api_reference/community/index.html) | ✅ | ❌ | ❌ |

| Source | Document Lazy Loading | Native Async Support

To access PyPDFium2 document loader you'll need to install the `langchain-community` integration package.

If you want to get automated best in-class tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:

# os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

# os.environ["LANGSMITH_TRACING"] = "true"

### Installation
Install **langchain_community**.

Now we can instantiate our model object and load documents:

from langchain_community.document_loaders import PyPDFium2Loader

Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'page': 0}, page_content='LayoutParser: A Unified Toolkit for Deep\r\nLearning Based Document Image Analysis\r\nZejiang Shen\r\n1\r\n(), Ruochen Zhang\r\n2\r\n, Melissa Dell\r\n3\r\n, Benjamin Charles Germain\r\nLee\r\n4\r\n, Jacob Carlson\r\n3\r\n, and Weining Li\r\n5\r\n1 Allen Institute for AI\r\nshannons@allenai.org 2 Brown University\r\nruochen zhang@brown.edu 3 Harvard University\r\n{melissadell,jacob carlson}@fas.harvard.edu\r\n4 University of Washington\r\nbcgl@cs.washington.edu 5 University of Waterloo\r\nw422li@uwaterloo.ca\r\nAbstract. Recent advances in document image analysis (DIA) have been\r\nprimarily driven by the application of neural networks. Ideally, research\r\noutcomes could be easily deployed in production and extended for further\r\ninvestigation. However, various factors like loosely organized codebases\r\nand sophisticated model configurations complicate the easy reuse of im\x02portant innovations by a wide audience. Though there have been on-going\r\nefforts to improve reusability and simplify deep learning (DL) model\r\ndevelopment in disciplines like natural language processing and computer\r\nvision, none of them are optimized for challenges in the domain of DIA.\r\nThis represents a major gap in the existing toolkit, as DIA is central to\r\nacademic research across a wide range of disciplines in the social sciences\r\nand humanities. This paper introduces LayoutParser, an open-source\r\nlibrary for streamlining the usage of DL in DIA research and applica\x02tions. The core LayoutParser library comes with a set of simple and\r\nintuitive interfaces for applying and customizing DL models for layout de\x02tection, character recognition, and many other document processing tasks.\r\nTo promote extensibility, LayoutParser also incorporates a community\r\nplatform for sharing both pre-trained models and full document digiti\x02zation pipelines. We demonstrate that LayoutParser is helpful for both\r\nlightweight and large-scale digitization pipelines in real-word use cases.\r\nThe library is publicly available at https://layout-parser.github.io.\r\nKeywords: Document Image Analysis· Deep Learning· Layout Analysis\r\n· Character Recognition· Open Source library· Toolkit.\r\n1 Introduction\r\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\r\ndocument image analysis (DIA) tasks including document image classification [11,\r\narXiv:2103.15348v2 [cs.CV] 21 Jun 2021\n')

stdout

{'source': './example_data/layout-parser-paper.pdf', 'page': 0}

For detailed documentation of all PyPDFium2Loader features and configurations head to the API reference: https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.PyPDFium2Loader.html


This notebook covers how to load a document object from something you just want to copy and paste. In this case, you don't even need to use a DocumentLoader, but rather can just construct the Document directly.

from langchain_core.documents import Document

text = "..... put the text you copy pasted here......"

If you want to add metadata about the where you got this piece of text, you easily can with the metadata key.

metadata = {"source": "internet", "date": "Friday"}

doc = Document(page_content=text, metadata=metadata)


This loader loads all PDF files from a specific directory.

| Class | Package | Local | Serializable | JS support|

| [PyPDFDirectoryLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.PyPDFDirectoryLoader.html) | [langchain_community](https://python.langchain.com/api_reference/community/index.html) | ✅ | ❌ | ❌ |

| Source | Document Lazy Loading | Native Async Support

No credentials are needed for this loader.

If you want to get automated best in-class tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:

# os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

# os.environ["LANGSMITH_TRACING"] = "true"

### Installation
Install **langchain_community**.

Now we can instantiate our model object and load documents:

from langchain_community.document_loaders import PyPDFDirectoryLoader

Document(metadata={'source': 'example_data/layout-parser-paper.pdf', 'page': 0}, page_content='LayoutParser : A Uniﬁed Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1( \x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1Allen Institute for AI\nshannons@allenai.org\n2Brown University\nruochen zhang@brown.edu\n3Harvard University\n{melissadell,jacob carlson }@fas.harvard.edu\n4University of Washington\nbcgl@cs.washington.edu\n5University of Waterloo\nw422li@uwaterloo.ca\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model conﬁgurations complicate the easy reuse of im-\nportant innovations by a wide audience. Though there have been on-going\neﬀorts to improve reusability and simplify deep learning (DL) model\ndevelopment in disciplines like natural language processing and computer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademic research across a wide range of disciplines in the social sciences\nand humanities. This paper introduces LayoutParser , an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitive interfaces for applying and customizing DL models for layout de-\ntection, character recognition, and many other document processing tasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases.\nThe library is publicly available at https://layout-parser.github.io .\nKeywords: Document Image Analysis ·Deep Learning ·Layout Analysis\n·Character Recognition ·Open Source library ·Toolkit.\n1 Introduction\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\ndocument image analysis (DIA) tasks including document image classiﬁcation [ 11,arXiv:2103.15348v2  [cs.CV]  21 Jun 2021')

{'source': 'example_data/layout-parser-paper.pdf', 'page': 0}

For detailed documentation of all PyPDFDirectoryLoader features and configurations head to the API reference: https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.pdf.PyPDFDirectoryLoader.html


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2dfc4698",
   "metadata": {},
   "source": [
    "# URL\n",
    "\n",
    "This example covers how to load `HTML` documents from a list of `URLs` into the `Document` format that we can use downstream.\n",
    "\n",
    "## Unstructured URL Loader\n",
    "\n",
    "For the examples below, please install the `unstructured` library and see [this guide](/docs/integrations/providers/unstructured/) for more instructions on setting up Unstructured locally, including setting up required system dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb26084d-a2b0-4685-9ec4-346139ffe0fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet unstructured"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "16c3699e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredURLLoader\n",
    "\n",
    "urls = [\n",
    "    \"https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023\",\n",
    "    \"https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33089aba-ff74-4d00-8f40-9449c29587cc",
   "metadata": {},
   "source": [
    "Pass in ssl_verify=False with headers=headers to get past ssl_verification errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "00f46fda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [


{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "20deed05",
   "metadata": {},
   "source": [
    "# Unstructured\n",
    "\n",
    "This notebook covers how to use `Unstructured` [document loader](https://python.langchain.com/docs/concepts/#document-loaders) to load files of many types. `Unstructured` currently supports loading of text files, powerpoints, html, pdfs, images, and more.\n",
    "\n",
    "Please see [this guide](../../integrations/providers/unstructured.mdx) for more instructions on setting up Unstructured locally, including setting up required system dependencies.\n",
    "\n",
    "## Overview\n",
    "### Integration details\n",
    "\n",
    "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/docs/integrations/document_loaders/file_loaders/unstructured/)|\n",
    "| :--- | :--- | :---: | :---: |  :---: |\n",
    "| [UnstructuredLoader](https://python.langchain.com/api_reference/unstructured/document_loaders/langchain_unstructured.document_loaders.UnstructuredLoader.html) | [langchain_unstructured](https://python.langchain.com/api_reference/unstructured/index.html) | ✅ | ❌ | ✅ | \n",
    "### Loader features\n",
    "| Source | Document Lazy Loading | Native Async Support\n",
    "| :---: | :---: | :---: | \n",
    "| UnstructuredLoader | ✅ | ❌ | \n",
    "\n",
    "## Setup\n",
    "\n",
    "### Credentials\n",
    "\n",
    "By default, `langchain-unstructured` installs a smaller footprint that requires offloading of the partitioning logic to the Unstructured API, which requires an API key. If you use the local installation, you do not need an API key. To get your API key, head over to [this site](https://unstructured.io) and get an API key, and then set it in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2886982e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "if \"UNSTRUCTURED_API_KEY\" not in os.environ:\n",
    "    os.environ[\"UNSTRUCTURED_API_KEY\"] = getpass.getpass(\n",
    "        \"Enter your Unstructured API key: \"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e75e2a6d",
   "metadata": {},
   "source": [
    "### Installation\n",
    "\n",
    "#### Normal Installation\n",
    "\n",
    "The following packages are required to run the rest of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9de83b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install package, compatible with API partitioning\n",
    "%pip install --upgrade --quiet langchain-unstructured unstructured-client unstructured \"unstructured[pdf]\" python-magic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "637eda35",
   "metadata": {},
   "source": [
    "#### Installation for Local\n",
    "\n",
    "If you would like to run the partitioning logic locally, you will need to install a combination of system dependencies, as outlined in the [Unstructured documentation here](https://docs.unstructured.io/open-source/installation/full-installation).\n",
    "\n",
    "For example, on Macs you can install the required dependencies with:\n",
    "\n",
    "```bash\n",
    "# base dependencies\n",
    "brew install libmagic poppler tesseract\n",
    "\n",
    "# If parsing xml / html documents:\n",
    "brew install libxml2 libxslt\n",
    "```\n",
    "\n",
    "You can install the required `pip` dependencies needed for local with:\n",
    "\n",
    "```bash\n",
    "pip install \"langchain-unstructured[local]\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9c1c775",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "\n",
    "The `UnstructuredLoader` allows loading from a variety of different file types. To read all about the `unstructured` package please refer to their [documentation](https://docs.unstructured.io/open-source/introduction/overview)/. In this example, we show loading from both a text file and a PDF file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "79d3e549",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_unstructured import UnstructuredLoader\n",
    "\n",
    "file_paths = [\n",
    "    \"./example_data/layout-parser-paper.pdf\",\n",
    "    \"./example_data/state_of_the_union.txt\",\n",
    "]\n",
    "\n",
    "\n",
    "loader = UnstructuredLoader(file_paths)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b68dcab",
   "metadata": {},
   "source": [
    "## Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8da59ef8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO: pikepdf C++ to Python logger bridge initialized\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Document(metadata={'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}, page_content='1 2 0 2')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs = loader.load()\n",
    "\n",
    "docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "97f7aa1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': './example_data/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': './example_data', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-02-27T15:49:27', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}\n"
     ]
    }
   ],
   "source": [
    "print(docs[0].metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d7f991b",
   "metadata": {},
   "source": [
    "## Lazy Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b05604d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [


"import requests\n",
    "from langchain_unstructured import UnstructuredLoader\n",
    "from unstructured_client import UnstructuredClient\n",
    "from unstructured_client.utils import BackoffStrategy, RetryConfig\n",
    "\n",
    "client = UnstructuredClient(\n",
    "    api_key_auth=os.getenv(\n",
    "        \"UNSTRUCTURED_API_KEY\"\n",
    "    ),  # Note: the client API param is \"api_key_auth\" instead of \"api_key\"\n",
    "    client=requests.Session(),  # Define your own requests session\n",
    "    server_url=\"https://api.unstructuredapp.io/general/v0/general\",  # Define your own api url\n",
    "    retry_config=RetryConfig(\n",
    "        strategy=\"backoff\",\n",
    "        retry_connection_errors=True,\n",
    "        backoff=BackoffStrategy(\n",
    "            initial_interval=500,\n",
    "            max_interval=60000,\n",
    "            exponent=1.5,\n",
    "            max_elapsed_time=900000,\n",
    "        ),\n",
    "    ),  # Define your own retry config\n",
    ")\n",
    "\n",
    "loader = UnstructuredLoader(\n",
    "    \"./example_data/layout-parser-paper.pdf\",\n",
    "    partition_via_api=True,\n",
    "    client=client,\n",
    "    split_pdf_page=True,\n",
    "    split_pdf_page_range=[1, 10],\n",
    ")\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "print(docs[0].metadata[\"filename\"], \": \", docs[0].page_content[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c66fbeb3",
   "metadata": {},
   "source": [
    "## Chunking\n",
    "\n",
    "The `UnstructuredLoader` does not support `mode` as parameter for grouping text like the older\n",
    "loader `UnstructuredFileLoader` and others did. It instead supports \"chunking\". Chunking in\n",
    "unstructured differs from other chunking mechanisms you may be familiar with that form chunks based\n",
    "on plain-text features--character sequences like \"\\n\\n\" or \"\\n\" that might indicate a paragraph\n",
    "boundary or list-item boundary. Instead, all documents are split using specific knowledge about each\n",
    "document format to partition the document into semantic units (document elements) and we only need to\n",
    "resort to text-splitting when a single element exceeds the desired maximum chunk size. In general,\n",
    "chunking combines consecutive elements to form chunks as large as possible without exceeding the\n",
    "maximum chunk size. Chunking produces a sequence of CompositeElement, Table, or TableChunk elements.\n",
    "Each “chunk” is an instance of one of these three types.\n",
    "\n",
    "See this [page](https://docs.unstructured.io/open-source/core-functionality/chunking) for more\n",
    "details about chunking options, but to reproduce the same behavior as `mode=\"single\"`, you can set\n",
    "`chunking_strategy=\"basic\"`, `max_characters=<some-really-big-number>`, and `include_orig_elements=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e9f1c20d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of LangChain documents: 1\n",
      "Length of text in the document: 42772\n"
     ]
    }
   ],
   "source": [
    "from langchain_unstructured import UnstructuredLoader\n",
    "\n",
    "loader = UnstructuredLoader(\n",
    "    \"./example_data/layout-parser-paper.pdf\",\n",
    "    chunking_strategy=\"basic\",\n",
    "    max_characters=1000000,\n",
    "    include_orig_elements=False,\n",
    ")\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "print(\"Number of LangChain documents:\", len(docs))\n",
    "print(\"Length of text in the document:\", len(docs[0].page_content))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec3c22d-02cd-498b-921f-b839d1404f32",
   "metadata": {},
   "source": [
    "## Loading web pages\n",
    "\n",
    "`UnstructuredLoader` accepts a `web_url` kwarg when run locally that populates the `url` parameter of the underlying Unstructured [partition](https://docs.unstructured.io/open-source/core-functionality/partitioning). This allows for the parsing of remotely hosted documents, such as HTML web pages.\n",
    "\n",
    "Example usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bf9a8546-659d-4861-bff2-fdf1ad93ac65",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Example Domain' metadata={'category_depth': 0, 'languages': ['eng'], 'filetype': 'text/html', 'url': 'https://www.example.com', 'category': 'Title', 'element_id': 'fdaa78d856f9d143aeeed85bf23f58f8'}\n",
      "\n",
      "page_content='This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.' metadata={'languages': ['eng'], 'parent_id': 'fdaa78d856f9d143aeeed85bf23f58f8', 'filetype': 'text/html', 'url': 'https://www.example.com', 'category': 'NarrativeText', 'element_id': '3652b8458b0688639f973fe36253c992'}\n",
      "\n",
      "page_content='More information...' metadata={'category_depth': 0, 'link_texts': ['More information...'], 'link_urls': ['https://www.iana.org/domains/example'], 'languages': ['eng'], 'filetype': 'text/html', 'url': 'https://www.example.com', 'category': 'Title', 'element_id': '793ab98565d6f6d6f3a6d614e3ace2a9'}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from langchain_unstructured import UnstructuredLoader\n",
    "\n",
    "loader = UnstructuredLoader(web_url=\"https://www.example.com\")\n",
    "docs = loader.load()\n",
    "\n",
    "for doc in docs:\n",
    "    print(f\"{doc}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce01aa40",
   "metadata": {},
   "source": [
    "## API reference\n",
    "\n",
    "For detailed documentation of all `UnstructuredLoader` features and configurations head to the API reference: https://python.langchain.com/api_reference/unstructured/document_loaders/langchain_unstructured.document_loaders.UnstructuredLoader.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}