Building a Simple VLM-Based Multimodal Information Retrieval System with NVIDIA NIM
In today’s data-driven world, the ability to retrieve accurate information from even modest amounts of data is vital for developers seeking streamlined, effective solutions for quick deployments, prototyping, or experimentation. One of the key challenges in information retrieval is managing the diverse modalities in unstructured datasets, including text, PDFs, images, tables, audio, video, and so on.
Multimodal AI models address this challenge by simultaneously processing multiple data modalities, generating cohesive and comprehensive output in different forms.
NVIDIA NIM
microservices simplify the secure and reliable deployment of AI
foundation models
for language,
computer vision
, speech, biology, and more.
NIM microservices can be deployed on NVIDIA-accelerated infrastructure anywhere and expose industry-standard APIs for fast integration with applications and popular AI development frameworks, including LangChain and LlamaIndex.
This post helps you get started with building a
vision language model
(VLM) based, multimodal, information retrieval system capable of answering complex queries involving text, images, and tables. We walk you through deploying an application using
LangGraph,
the state-of-the-art
llama-3.2-90b-vision-instruct
VLM, the optimized
mistral-small-24B-instruct
large language model (LLM), and NVIDIA NIM for deployment.
This method of building simple information retrieval systems offers several advantages over traditional ones. The latest VLM NIM microservice enables enhanced contextual understanding by processing lengthy, complex visual documents without sacrificing coherence. The integration of
LangChain’s tool calling
enables the system to create tools, dynamically select and use external tools, and improve the precision of data extraction and interpretation from various sources.
This system is good for enterprise applications because it generates structured outputs, ensuring consistency and reliability in responses. For more information about the implementation steps of this system, see the
/NVIDIA/GenerativeAIExamples
GitHub repo.
A simple HTML multimodal retrieval pipeline
The system consists of the following pipelines:
Document ingestion and preprocessing:
Runs a VLM on the images and translates them into text.
Question-answering:
Enables the user to ask questions of the system.
Both pipelines integrate NVIDIA NIM and LangGraph to process and understand text, images, complex visualizations, and tables effectively.
Data ingestion and preprocessing pipeline
This stage parses documents to process text, images, and tables separately. Tables are first converted into images, and images are processed by the NVIDIA-hosted NIM microservice API endpoint for the llama-3.2-90b-vision-instruct VLM to generate descriptive text.
Next, in the document reconstruction step, the descriptive text is merged with the original text of the document, then summarized by an
LLM with long context modeling
capability. In this implementation, llama-3.2-90b-vision-instruct is also used as the LLM, although other LLMs such as mistral-small-24b-instruct can also be deployed.
Finally, the complete text, summaries, images, and their descriptions are stored in a NoSQL database, along with unique document identifiers.
Figure 1. Data ingestion and preprocessing pipeline
LLMs with long context modeling
can process entire documents without fragmentation, enhancing comprehension of the document in a single pass, and capturing relationships and nuances across longer spans of text, leading to more accurate information retrieval.
In contrast, traditional models may handle inputs of up to a few thousand tokens, requiring lengthy documents to be split into smaller chunks to fit within the model’s context window. This chunking process can disrupt coherence and context, making it more difficult to accurately retrieve and rank relevant information.
However, long context modeling presents challenges related to scalability and cost, which must be considered when trading off with higher accuracy.
QA pipeline
All document summaries and their identifiers are compiled into a large prompt. When a query is sent, a
LLM with long context modeling
(mistral-small-24b-instruct in this case) processes the question, evaluates the relevance of each summary to the query, and returns the identifiers of the most relevant documents.
Figure 2. Question-answering pipeline
Next, the most relevant documents are fed into an LLM with long context (mistral-small-24b-instruct). The model generates an answer to the query based on the textual content. If the model identifies that an image may contain pertinent information based on its descriptive text, an additional step is triggered: the original image and the user’s question are sent to the VLM (llama-3.2-90b-vision-instruct), which can provide an answer based on the actual visual content.
Finally, the system combines both textual and visual insights to deliver a comprehensive answer.
Structured outputs ensure that the data returned by the model conforms to a predefined format, making it easier to extract specific information and perform subsequent operations. In contrast, unstructured or variable outputs can introduce ambiguities and difficulties in parsing the model’s responses, hindering automation and integration with other systems.
Generating structured data from models typically requires carefully designed prompts to guide the model into responding in a particular format, such as JSON. However, ensuring consistent adherence to this structure can be challenging due to the models’ natural tendency to generate free-form text.
NVIDIA NIM now natively supports
capabilities for generating structured outputs
. This means that you can rely on built-in functionalities to ensure that the model’s responses are consistently formatted, reducing the need for complex prompt engineering.
Integrating NVIDIA NIM with LangGraph
NVIDIA NIM offers seamless compatibility with popular frameworks and the latest AI models for your applications. The implementation of the pipeline integrates NVIDIA NIM with
LangGraph
, a framework to build agentic applications to determine the control flow, which has been widely adopted by the developer community. To orchestrate the workflow of this pipeline, the graph mainly consists of two nodes:
Assistant node:
Serves as an agent responsible for managing the logic and decision-making process. It interacts with the user’s inputs and invokes the necessary tools.
Tools node:
A collection of tools that perform specific tasks required by the assistant.
Figure 3. Use LangGraph to build an agent for the pipeline
Assistant node
The assistant node is a primary agent that operates according to the workflow outlined in Figure 3. The code of the main agent can be found in the
/NVIDIA/GenerativeAIExamples
GitHub repo.
Here are the agent inputs:
Collection_name
: The set of documents on which to search.
Question
: The user’s question.
document_id
: (Optional) If provided, the agent skips the document ranking phase.
This is the agent process:
Document selection
: If
document_id
is not provided, the agent invokes the
find_best_document_id
tool, which identifies the most relevant document for the user’s question within the specified collection.
Question answering
: With
document_id
, the agent uses the
query_document
tool. This tool attempts to answer the question using the LLM (mistral-small-24b-instruct) based on the text and image descriptions within the document.
Image analysis
(if necessary): If the
query_document
tool indicates that the answer might be in an image (by returning an
image_hash
value), the agent invokes the
query_image
tool. This tool retrieves the actual image and uses a VLM to analyze the image and find the answer.
Tools node
We implemented three key tools for the agent to perform its tasks.
Find_best_document_id
: Identify the most relevant document for the user’s question when
document_id
is not provided. For more information, see the
/NVIDIA/GenerativeAIExamples
GitHub repo.
query_document
: Search for an answer within the specified document. If the answer may be in an image, it provides details to query the image. For more information, see the
/NVIDIA/GenerativeAIExamples
GitHub repo.
query_image
: Analyze the actual image using a VLM when the answer might be within the image content. For more information, see the
/NVIDIA/GenerativeAIExamples
.
Binding external tools with models
Tool calling
is a feature that enables language models to integrate and interact with external tools or functions based on the prompts that they receive. This mechanism enables a model to decide which tools to use and how to use them to accomplish specific tasks.
Tool binding
empowers models to extend their capabilities dynamically, selecting appropriate tools during execution to provide more accurate, context-aware responses.
Binding external tools is particularly crucial in agentic frameworks, where agents must choose the appropriate tools and provide the necessary arguments to perform tasks effectively. The benefits of binding external tools include the following:
Extended capabilities
: Models can perform complex operations such as calculations, data retrieval, or API calls, which go beyond mere text generation.
Dynamic tool selection
: The model can assess in real time which tools are most suitable for the task, improving efficiency and relevance.
Seamless integration
: NVIDIA NIM supports the integration of external tools, such as LangChain and LangGraph, with open community models such as Llama 3.3. You can adopt these advanced features without making significant changes to your existing systems.
In this implementation, use LangChain’s
@tool decorator
to create three tools, then use the
.bind_tools
method to bind the tools with models.
Defining structured outputs with Pydantic
By defining the output schema with
Pydantic
and guiding an LLM NIM microservice such as mistral-small-24b-instruct through precise prompts, you ensure that the responses are consistent, reliable, and easily consumable by other components within the system. This approach is essential when integrating the LLM into automated workflows and agent-based frameworks such as LangGraph.
Define the structure
The process begins by defining the structure of the output that you expect from the LLM using Pydantic. This guarantees that the data returned by the model is consistent and can be easily parsed for
downstream processing
.
from typing import List, Optional
from pydantic import BaseModel, Field

class Document(BaseModel):
"""
Represents a document with an identifier and its summary.
"""

id: str = Field(..., description="Hash identifier of the document")
summary: str = Field(..., description="The summary of the document as is")

class BestDocuments(BaseModel):
"""
Contains a list of the best documents to answer the question and their summaries.
"""
documents: List[Document] = Field(..., description="List of best documents")

class Answer(BaseModel):
"""
Represents the answer to the user's question.
"""
answer: str = Field(..., description="Answer to the question posed by the user")
Next, instruct the LLM to generate outputs that align with the defined Pydantic structures. This is achieved by incorporating specific instructions within the prompt and using LangChain’s
with_structured_output
method.
Define the prompt
The
prompt_document_expert
contains detailed instructions for the LLM, specifying the expected input format (Markdown with document summaries) and the required output format (JSON matching the
BestDocuments
schema).
from langchain.chat_models import ChatNVIDIA
from langchain.prompts import ChatPromptTemplate

# Initialize the LLM with desired parameters

llm = ChatNVIDIA(model="mistralai/mistral-small-24b-instruct
", temperature=0, max_tokens=3000)

# Define the prompt template for the document expert

prompt_document_expert = ChatPromptTemplate.from_messages(  
    [  
        (  
            "system",  
            f""" # Extract Best Document Identifier from list of summaries, based on a question coming from the user. You are an expert in getting insights of a document, based on its summaries and you are able to figure the best matches to the question in terms of the summary of the document.  
            Provide no more than 3 of these documents.            ## Format of the Input            - The input is a markdown file containing second level headers (##) with the chapter index in the form ## Document <document_id> where document_id is an integer pointing to the index of the document. After the document heading there is the summary of the document which is relevant to understand the content of the document.            ## Format of the output            - The output is going to be the list of the best documents indices and a few of the corresponding summaries that help to answer the question coming from the user.            ## Content            - Here is the input you can work on:  
                {{documents_context}}  
                """,  
        ),  
        (  
            "human",  
            "Can you tell me what are the most relevant document ids for this question: {question}"  
        ),  
        ("human", "Tip: Make sure to answer in the correct format"),  
    ]  
)
Prepare context
The
get_context
function prepares the input data by retrieving document summaries and formatting them appropriately.
def get_context(input_data: dict) -> dict:  
  
    collection_name = input_data.get("collection_name")  
    question = input_data.get("question")  
  
    documents_context = get_document_summaries_markdown(collection_name)  
  
    # print(context)  
    return {"documents_context": documents_context,  
            "collection_name": collection_name,  
            "question": question}
Bind the structured output
The
llm.with_structured_output(BestDocuments)
method instructs the LLM to produce output conforming to the
BestDocuments
Pydantic model. This method internally handles the parsing and validation of the LLM’s response, ensuring that the output matches the expected structure.
LangChain’s
with_structured_output
method simplifies the process of binding the model to produce structured outputs. It abstracts the complexity of parsing and validating the LLM’s responses, enabling you to focus on defining the desired output structure and the prompt instructions.
Finally, create a chain to process the input and generate the structured output:
chain_document_expert = (  
    RunnableLambda(get_context) | prompt_document_expert | llm.with_structured_output(BestDocuments) | (lambda x: x.dict())  
)
End-to-end tool in action
To get started with the multimodal retrieval system, clone the
/NVIDIA/GenerativeAIExamples
GitHub repo and follow the Quick Start guide to set up the service. When it’s up and running, open your web browser and navigate to
http://localhost:7860
to access the system through the Gradio user interface.
For example, explore how the system processes queries on the NVIDIA Technical Blog. Ask a question about a bar chart showing the NVIDIA H100 GPU performance from one of the posts. The
Select Question
field is for evaluation purposes, with the
Ground Truth Answer
field value provided by a human.
Figure 4. Agent multi-document evaluation
This system generates an accurate answer based on the bar chart and also displays the relevant image for reference, such as the chart showing RetinaNet achieving 54%. This ensures precise answers while enabling users to visually verify the referenced data.
Figure 5. Agent result with source graph for verification
Video 1. How to Insert HTML Documents into a Multimodal Retriever Collection Using NVIDIA NIM
Video 2. How to Search Text and Images Within a Multimodal Retriever Collection Using NVIDIA NIM
Challenges and solutions
As data volumes increase, so does the complexity of processing and retrieving relevant information. Handling large datasets efficiently is essential to maintaining performance and ensuring user satisfaction. In this information retrieval system, the sheer amount of document summaries can exceed the context window of even long-context models, making it challenging to process all summaries in a single prompt.
Processing large volumes of data also demands considerable computational resources, which can result in higher costs and increased latency. Optimizing resource utilization is crucial to delivering fast and accurate responses while minimizing unnecessary expenses.
Hierarchical document reranking solution
To address scalability challenges, we implemented a hierarchical approach in the initial document reranking phase. Instead of processing all document summaries simultaneously, we divided them into manageable batches that fit within the model’s context window. The process involves multiple stages:
Batch processing
: Summaries are grouped into batches that the model can handle without exceeding the prompt size limitations.
Intermediate reranking
: The model evaluates each batch separately, ranking the documents within each group.
Selection of top candidates
: The most relevant documents from each batch are selected to proceed to the next stage.
Final reranking
: The top candidates from all batches are combined and re-evaluated to identify the most relevant document.
Considering both scalability and cost concerns, this hierarchical approach ensures that all documents are considered without exceeding the model’s capacity. It not only improves scalability, but also boosts efficiency by narrowing down the candidate documents systematically until the most relevant one is identified.
Future prospects with smaller models
Using language models, especially those with long-context capabilities, involves processing a large number of tokens, which can incur significant costs. Each token processed adds to the overall expense, making cost management a critical consideration when deploying these systems at scale.
The concern about cost is indeed valid. However, the landscape of language models is rapidly evolving, with smaller models becoming increasingly capable and efficient. As these advancements continue, these smaller models may offer similar performance at a fraction of the cost.
Conclusion
This post discussed the implementation of a simple multimodal information retrieval pipeline that uses NVIDIA NIM and LangGraph. The pipeline offers several advantages over existing information retrieval methods:
Enhanced comprehension of documents
A multimodal model to extract information from images, tables, and text
Seamless integration of external tools
Generation of consistent and structured output
Using NVIDIA NIM and LangGraph, you can build on this work and customize it to suit specific needs. To get started, you can find source code in the
/NVIDIA/GenerativeAIExamples
GitHub repo.
NVIDIA NIM
also offers access to more models optimized for NVIDIA GPUs. You can explore
NVIDIA NeMo
, a scalable
generative AI
framework designed for researchers and PyTorch developers working on LLMs, multimodal models, and more.
If you are working with a large corpora of enterprise data and are looking to develop enterprise-ready, real-time
multilingual and cross-lingual information retrieval systems
to generate context-aware responses, learn more about
NVIDIA NeMo Retriever
.
使用 NVIDIA NIM 构建基于 VLM 的简单多模态信息检索系统
在当今数据驱动的世界中，即使是从少量数据中检索准确信息的能力，对于寻求精简、有效的快速部署、原型设计或实验解决方案的开发者来说也至关重要。信息检索领域的主要挑战之一是管理非结构化数据集中的各种模式，包括文本、PDF、图像、表格、音频、视频等。
多模态 AI 模型通过同时处理多个数据模式来应对这一挑战，以不同的形式生成连贯一致的全面输出。
NVIDIA NIM
微服务可简化 AI
基础模型
在语言、
计算机视觉
、语音、生物学等领域的安全可靠部署。
NIM 微服务可随时随地部署在 NVIDIA 加速基础设施上，并提供行业标准 API，以快速集成应用和热门 AI 开发框架 (包括 LangChain 和 LlamaIndex)。
本文将帮助您着手构建基于
视觉语言模型
（VLM）的多模态信息检索系统，该系统能够回答涉及文本、图像和表格的复杂查询。我们将引导您使用
LangGraph
部署应用程序、先进的
llama-3.2-90b-vision-instruct
VLM、经过优化的
mistral-small-24B-instruct
大语言模型（LLM），以及用于部署的 NVIDIA NIM。
与传统方法相比，这种构建简单信息检索系统的方法具有许多优势。最新的 VLM NIM 微服务可在不牺牲一致性的情况下处理冗长而复杂的视觉文档，从而增强上下文理解。通过集成
LangChain 的工具调用
，系统能够创建工具，动态选择和使用外部工具，并提高从各种来源提取和解释数据的精度。
此系统适用于企业应用，因为它生成结构化输出，确保响应的一致性和可靠性。有关此系统的实施步骤的更多信息，请参阅
/NVIDIA/GenerativeAIExamples
Github 仓库。
简单的 HTML 多模态检索工作流
该系统由以下管道组成：
文档提取和预处理：在图像上运行 VLM 并将其转换为文本。
问答：允许用户提出系统问题。
这两个工作流均集成了 NVIDIA NIM 和 LangGraph，可有效处理和理解文本、图像、复杂的可视化效果和表格。
数据提取和预处理 pipeline
此阶段会解析文档，分别处理文本、图像和表格。首先将表格转换为图像，然后由 NVIDIA 托管的 NIM 微服务 API 端点为 llama-3.2-90b-vision-instruct VLM 处理图像，以生成描述性文本。
接下来，在文档重建步骤中，描述性文本将与文档的原始文本合并，然后由具有长上下文建模功能的 LLM 进行汇总。在此实施中，还可将 Llama-3.2-90b-vision-instruct 用作 LLM，不过也可部署其他 LLM（例如 mistral-small-24b-instruct）。
最后，完整的文本、摘要、图像及其说明将存储在 NoSQL 数据库中，以及唯一的文档标识符。 图 1. 数据提取和预处理管道
采用长上下文建模的 LLMs 可以处理整个文档，而不会出现碎片，从而在单个通道中增强对文档的理解，并捕获更长的文本跨度中的关系和细微差别，从而实现更准确的信息检索。
相比之下，传统模型可能会处理多达数千个 tokens 的输入，因此需要将冗长的文档拆分成较小的块，以适应模型的上下文窗口。这种分块过程会破坏一致性和上下文，使准确检索相关信息并对其进行排名变得更加困难。
但是，长上下文建模会带来与可扩展性和成本相关的挑战，在以更高的准确性进行权衡时必须考虑这些挑战。
QA 管道
所有文档摘要及其标识符都会编译成一个大型提示。发送查询时，使用长上下文建模（本例中为 mistral-small-24b-instruct）的 LLM 会处理问题，评估每个摘要与查询的相关性，并返回最相关文档的标识符。 图 2、问题回答管道
接下来，将最相关的文档输入到具有长上下文 (mistral-small-24b-instruct) 的 LLM 中。模型会根据文本内容生成查询答案。如果模型根据描述性文本识别出图像可能包含相关信息，则会触发另一个步骤：将原始图像和用户的问题发送至 VLM (llama-3.2-90b-vision-instruct)，VLM 可以根据实际视觉内容提供答案。
最后，该系统将文本和视觉见解相结合，以提供全面的答案。
结构化输出可确保模型返回的数据符合预定义的格式，从而更轻松地提取特定信息并执行后续运算。相比之下，非结构化或可变输出会在解析模型的响应时引入模糊性和困难，从而阻碍自动化以及与其他系统的集成。
从模型生成结构化数据通常需要精心设计的提示，以指导模型以特定格式（例如 JSON）做出响应。但是，由于模型自然倾向于生成自由格式文本，因此确保一致性遵循此结构可能具有挑战性。
NVIDIA NIM 现在原生支持生成结构化输出的功能。这意味着，您可以依靠内置功能来确保模型的响应格式保持一致，从而减少对复杂提示工程的需求。
将 NVIDIA NIM 与 LangChain 集成
NVIDIA NIM 为您的应用提供与热门框架和最新 AI 模型的无缝兼容性。该流程的实施将 NVIDIA NIM 与 LangChain 相集成。LangChain 是一个用于构建代理应用以确定控制流的框架，已被开发者社区广泛采用。为编排此工作流的工作流，该图形主要由两个节点组成：
助理节点：充当负责管理逻辑和决策过程的代理。它与用户的输入进行交互，并调用必要的工具。
工具节点：用于执行助手所需特定任务的工具集合。
图 3、使用 LangGraph 为管道构建代理
助理节点
助手节点是根据图 3 中概述的工作流程运行的主代理。主代理的代码可在
/NVIDIA/GenerativeAIExamples
GitHub repo 中找到。
智能体输入如下：
Collection_name
：要在其上搜索的文档集。
Question
：用户的问题。
document_id
：(可选) 如果提供，代理会跳过文档排名阶段。
这是智能体流程：
文档选择
：如果未提供
document_id
，代理会调用
find_best_document_id
工具，该工具可在指定集合中识别与用户问题最相关的文档。
问题回答：对于
document_id
，代理使用
query_document
工具。此工具会尝试使用 LLM (mistral-small-24b-instruct) 根据文档中的文本和图像描述来回答此问题。
图像分析 (如有必要)：如果
query_document
工具表明答案可能在图像中 (通过返回
image_hash
值)，代理会调用
query_image
工具。此工具会检索实际图像，并使用 VLM 分析图像并寻找答案。
工具节点
我们为智能体实施了三个关键工具来执行任务。
Find_best_document_id
：在未提供
document_id
时，确定与用户问题最相关的文档。有关更多信息，请参阅
/NVIDIA/GenerativeAIExamples
Github 存储库。
query_document
：在指定文档中搜索答案。如果答案可能在图像中，则会提供查询图像所需的详细信息。有关更多信息，请参阅
/NVIDIA/GenerativeAIExamples
GitHub 存储库。
query_image
：当答案可能在图像内容中时，使用 VLM 分析实际图像。有关更多信息，请参阅/
NVIDIA/GenerativeAIExamples
。
将外部工具与模型绑定
工具调用是一项功能，可让语言模型根据收到的提示集成外部工具或函数并与之交互。此机制使模型能够决定使用哪些工具以及如何使用这些工具来完成特定任务。
工具绑定使模型能够动态扩展其功能，在执行期间选择合适的工具，以提供更准确的上下文感知响应。
绑定外部工具在代理框架中尤为重要，在这种框架中，代理必须选择合适的工具并提供有效执行任务所需的参数。绑定外部工具的优势包括：
扩展功能
：模型可以执行计算、数据检索或 API 调用等复杂操作，而不仅仅是文本生成。
动态工具选择
：模型可以实时评估哪些工具最适合任务，从而提高效率和相关性。
无缝集成：NVIDIA NIM 支持将 LangChain 和 LangGraph 等外部工具与 Llama 3.3 等开放式社区模型集成。您可以采用这些高级功能，而无需对现有系统进行重大更改。
在此实现中，使用 LangChain 的
@tool
装饰器创建三个工具，然后使用
.bind_tools
方法将这些工具与模型绑定。
使用 PyTorch 定义结构化输出
通过使用
Pydantic
定义输出模式，并通过精确的提示引导 LLM NIM 微服务 (例如 mistral-small-24b-instruct) ，您可以确保响应一致、可靠，并且易于被系统中的其他组件使用。当将 LLM 集成到自动化工作流和基于代理的框架 (例如 LangChain) 时，这种方法至关重要。
定义结构
首先，使用 Pydantic 定义 LLM 的预期输出结构。这可确保模型返回的数据保持一致，并可轻松解析以进行下游处理。
from typing import List, Optional
from pydantic import BaseModel, Field

class Document(BaseModel):
"""
Represents a document with an identifier and its summary.
"""

id: str = Field(..., description="Hash identifier of the document")
summary: str = Field(..., description="The summary of the document as is")

class BestDocuments(BaseModel):
"""
Contains a list of the best documents to answer the question and their summaries.
"""
documents: List[Document] = Field(..., description="List of best documents")

class Answer(BaseModel):
"""
Represents the answer to the user's question.
"""
answer: str = Field(..., description="Answer to the question posed by the user")
接下来，指示 LLM 生成与定义的 Pydantic 结构保持一致的输出。这是通过在提示符中加入特定指令并使用 LangChain 的
with_structured_output
方法实现的。
定义提示
prompt_document_expert
包含 LLM 的详细说明，可指定预期的输入格式 (带有文档摘要的 Markdown) 和所需的输出格式 (与
BestDocuments
架构匹配的 JSON)。
from langchain.chat_models import ChatNVIDIA
from langchain.prompts import ChatPromptTemplate

# Initialize the LLM with desired parameters

llm = ChatNVIDIA(model="mistralai/mistral-small-24b-instruct
", temperature=0, max_tokens=3000)

# Define the prompt template for the document expert

prompt_document_expert = ChatPromptTemplate.from_messages(  
    [  
        (  
            "system",  
            f""" # Extract Best Document Identifier from list of summaries, based on a question coming from the user. You are an expert in getting insights of a document, based on its summaries and you are able to figure the best matches to the question in terms of the summary of the document.  
            Provide no more than 3 of these documents.            ## Format of the Input            - The input is a markdown file containing second level headers (##) with the chapter index in the form ## Document <document_id> where document_id is an integer pointing to the index of the document. After the document heading there is the summary of the document which is relevant to understand the content of the document.            ## Format of the output            - The output is going to be the list of the best documents indices and a few of the corresponding summaries that help to answer the question coming from the user.            ## Content            - Here is the input you can work on:  
                {{documents_context}}  
                """,  
        ),  
        (  
            "human",  
            "Can you tell me what are the most relevant document ids for this question: {question}"  
        ),  
        ("human", "Tip: Make sure to answer in the correct format"),  
    ]  
)
准备上下文
get_context
函数通过检索文档摘要并对其进行适当格式化来准备输入数据。
def get_context(input_data: dict) -> dict:  
  
    collection_name = input_data.get("collection_name")  
    question = input_data.get("question")  
  
    documents_context = get_document_summaries_markdown(collection_name)  
  
    # print(context)  
    return {"documents_context": documents_context,  
            "collection_name": collection_name,  
            "question": question}
绑定结构化输出
llm.with_structured_output(BestDocuments)
方法指示 LLM 生成符合
BestDocuments
Pydantic 模型的输出。此方法在内部处理 LLM 响应的解析和验证，确保输出与预期结构相匹配。
LangChain 的
with_structured_output
方法简化了绑定模型以生成结构化输出的过程。它抽象化了解析和验证 LLM 响应的复杂性，使您能够专注于定义所需的输出结构和提示指令。
最后，创建一个链来处理输入并生成结构化输出：
chain_document_expert = (  
    RunnableLambda(get_context) | prompt_document_expert | llm.with_structured_output(BestDocuments) | (lambda x: x.dict())  
)
端到端工具的实际应用
要开始使用多模态检索系统，请克隆
/NVIDIA/GenerativeAIExamples
GitHub 存储库，然后按照快速入门指南设置服务。在服务启动并运行时，打开 Web 浏览器并导航至
http://localhost:7860
，通过 Gradio 用户界面访问系统。
例如，在 NVIDIA 技术博客上探索系统如何处理查询。在其中一篇博文中，您可以询问有关显示 NVIDIA H100 GPU 性能的条形图的问题。“
Select Question
” 字段用于评估，真值答案字段值由人类提供。 图 4、Agent 多文档评估
该系统会根据条形图生成准确的答案，并显示相关图像以供参考，例如图表显示 RetinaNet 达到了 54%。这可确保准确的答案，同时使用户能够以直观方式验证引用数据。 图 5、Agent 结果与用于验证的源图形
视频1. 如何使用 NVIDIA NIM 将 HTML 文档插入多模态检索器集合
视频2. 如何使用 NVIDIA NIM 在多模态检索器集合中搜索文本和图像
挑战和解决方案
随着数据量的增加，处理和检索相关信息的复杂性也随之增加。高效处理大型数据集对于保持性能和确保用户满意度至关重要。在此信息检索系统中，文档摘要的数量甚至可能超过长上下文模型的上下文窗口，这使得在单个提示中处理所有摘要具有挑战性。
处理大量数据还需要大量计算资源，这可能会导致成本增加和延迟增加。优化资源利用率对于提供快速准确的响应，同时最大限度地减少不必要的支出至关重要。
分层文档重新排序解决方案
为应对可扩展性挑战，我们在初始文档重新排序阶段实施了分层方法。我们不会同时处理所有文档摘要，而是将其分为可管理的批量，以适应模型的上下文窗口。此过程涉及多个阶段：
批量处理
：将摘要分组为模型可以处理的批量，且不会超过提示大小限制。
中级重新排序
：模型分别评估每个批次，对每个组中的文档进行排序。
选择最优秀的候选文档
：从每个批次中选择最相关的文档，以进入下一阶段。
最终重新排名
：系统会对所有批次中排名靠前的候选文档进行合并和重新评估，以确定相关性最高的文档。
考虑到可扩展性和成本问题，这种分层方法可确保在不超出模型容量的情况下考虑所有文档。它不仅提高了可扩展性，而且还通过系统缩小候选文档的范围来提高效率，直到识别出最相关的文档。
小型模型的未来前景
使用语言模型，尤其是具有长上下文功能的语言模型，涉及处理大量 token，而这可能会产生巨大的成本。处理的每个 token 都会增加总支出，因此在大规模部署这些系统时，成本管理是一个重要考虑因素。
对成本的担心确实是站得住脚的。然而，语言模型的格局正在迅速演变，小型模型的功能和效率也在不断提升。随着这些进步的继续，这些较小的模型可能以远低于成本提供相似的性能。
结束语
本文讨论了如何使用 NVIDIA NIM 和 LangChain 实现简单的多模态信息检索工作流。与现有的信息检索方法相比，Pipeline 具有以下优势：
增强对文档的理解
用于从图像、表格和文本中提取信息的多模态模型
无缝集成外部工具
生成一致的结构化输出
借助 NVIDIA NIM 和 LangGraph，您可以在此基础上进行构建并对其进行定制，以满足特定需求。首先，您可以在
/NVIDIA/GenerativeAIExamples
GitHub repo 中找到源代码。
NVIDIA NIM
还支持访问更多针对 NVIDIA GPU 优化的模型。您可以探索
NVIDIA NeMo
，这是一个可扩展的生成式
AI
框架，专为研究 LLM、多模态模型等的研究人员和 PyTorch 开发者而设计。
如果您正在处理大型企业数据语料库，并希望开发企业就绪的实时多语种和跨语言信息检索系统来生成上下文感知响应，请详细了解
NVIDIA NeMo Retriever
。