Build a Generative AI Medical Device Training Assistant with NVIDIA NIM Microservices
Innovation in medical devices continues to accelerate, with a record number authorized by the
FDA every year
. When these new or updated devices are introduced to clinicians and patients, they require training to use them properly and safely.
Once in use, clinicians or patients may need help troubleshooting issues. Medical devices are often accompanied by lengthy and technically complex Instructions for Use (IFU) manuals, which describe the correct use of the device. It can be difficult to find the right information quickly and training on a new device is a time-consuming task. Medical device representatives often provide support training, but may not be present to answer all questions in real time. These issues can cause delays in using medical devices and adopting newer technologies, and in some cases, lead to incorrect usage.
Using generative AI for troubleshooting medical devices
Retrieval-augmented generation (RAG) uses deep learning models, including large language models (LLMs),  for efficient search and retrieval of information using natural language. Using RAG, users can receive easy-to-understand instructions for specific questions in a large text corpus, such as in an IFU. Speech AI models, such as automatic speech recognition (ASR) and text-to-speech (TTS) models, enable users to communicate with these advanced generative AI workflows using their voice, which is important in sterile environments like the operating room.
NVIDIA NIM inference microservices are GPU-optimized and highly performant containers for these models that provide the lowest total cost of ownership and the best inference optimization for the latest models. By integrating RAG and speech AI with the efficiency and simplicity of deploying NIM microservices, companies developing advanced medical devices can provide clinicians with accurate, hands-free answers in real time.
Figure 1. The chatbot user interface of the medical device training assistant
A medical device training assistant built with NIM microservices
In this tutorial, we build a RAG pipeline with optional speech capabilities to answer questions about a medical device using its IFU. The code used is available on
GitHub
.
We use the following NIM microservices in our RAG pipeline. You have the flexibility to change the components in the pipeline to other NIM microservices for different models:
Llama3 70B Instruct
(meta/llama3-70b-instruct): A large language model that generates the answer to the user question based on the retrieved text.
NV-EmbedQA-e5-v5
(nvidia/nv-embedqa-e5-v5): An embedding model that embeds the text chunks from the IFU and the queries from the user.
NV-RerankQA-Mistral-4b-v3
(nvidia/nv-rerankqa/mistral-4b-v3): A reranking model that reranks the retrieved text chunks for the text generation step by the LLM.
RIVA ASR
: An automatic speech recognition model that transcribes the user’s speech query into text for the model.
RIVA TTS
: The text-to-speech model that outputs the audio of the response from the LLM.
RAG has two steps: document ingestion, then retrieval and generation of answers. These steps and the associated NIM microservices can be found in the reference architecture diagram in Figure 2.
Figure 2. The reference architecture shows document ingestion and retrieval
Using NVIDIA NIM
You can access NIM microservices by signing up for free API credits on the API Catalog at
build.nvidia.com
or by deploying on your own compute infrastructure.
In this tutorial, we use the API Catalog endpoints. More information on using NIM microservices, finding your API key, and other prerequisites can be found on
GitHub
.
Follow these steps to build a RAG pipeline with optional speech for answering medical device questions using its IFU.
Build and start the containers
See the docker compose files we’ve created to launch the containers with the NIM microservices and vector database.
Detailed instructions
and code can be accessed on GitHub.
Ingest the device manual
Navigate your browser to upload your IFU in the “Knowledge Base” tab as shown in Figure 3.
Figure 3. The document ingestion page of the medical device training assistant
Retrieve and generate answers
Navigate to the “Converse” tab to begin the conversation with the IFU (Figure 1). Make sure to click “Use Knowledge Base” to use the IFU as a knowledge resource.
To use speech to converse, click the microphone next to the text input area, and the RIVA ASR model will transcribe your question. To receive speech as an output, click the “Enable TTS output”. More information about using and troubleshooting the UI is on the
GitHub documentation
.
Evaluate on a custom dataset
Evaluate the performance of the RAG pipeline using a custom dataset of questions and automated RAGAS metrics. RAGAS metrics evaluate the performance of both the retriever and generator and are a common method for evaluating RAG pipelines in an automated fashion. Instructions on how to use the evaluation script are on
GitHub
.
Getting started
To get started with this workflow, visit the
GenerativeAIExamples GitHub repository
, which contains all of the code used in this tutorial as well as extensive documentation.
For more information on NIM microservices, you can learn more from the
official NIM documentation
and ask questions on our NVIDIA Developer
NIM Forum
.
利用 NVIDIA NIM 微服务构建生成式 AI 医疗设备训练助手
医疗设备创新不断加速，每年获得 FDA 批准的数量都创下新高。向临床医生和患者介绍这些全新或更新的设备时，他们需要接受培训，以正确、安全地使用这些设备。
使用后，临床医生或患者可能需要获得排查问题的帮助。医疗设备通常附有冗长且技术复杂的 Instructions for Use (IFU) 手册，这些手册描述了设备的正确用法。快速找到正确的信息可能很困难，在新设备上训练是一项耗时的任务。医疗设备代表通常会提供支持培训，但可能无法实时回答所有问题。这些问题可能会延迟医疗设备的使用和较新技术的采用，在某些情况下，还会导致错误使用。
使用生成式 AI 对医疗设备进行故障排除
检索增强型生成 (RAG) 使用深度学习模型 (包括大语言模型 (LLMs))，使用自然语言高效搜索和检索信息。借助 RAG，用户可以接收大型文本语料库 (例如 IFU) 中特定问题的易于理解的说明。语音 AI 模型，如自动语音识别 (ASR) 和文本转语音 (TTS) 模型，使用户能够使用语音与这些先进的生成式 AI 工作流进行通信，这在手术室等无菌环境中至关重要。
NVIDIA NIM 推理微服务是为这些模型优化的 GPU 高性能容器，可为最新模型提供最低的总体拥有成本和最佳的推理优化。通过将 RAG 和语音 AI 与部署 NIM 微服务的效率和简便性相集成，开发先进医疗设备的公司可以实时为临床医生提供准确、免持的答案。
图 1、医疗设备训练助手的聊天机器人用户界面
使用 NIM 微服务构建的医疗设备训练助手
在本教程中，我们构建了具有可选语音功能的 RAG 工作流，以回答有关使用其 IFU 的医疗设备的问题。所使用的代码可在 GitHub 上获取。
我们在 RAG 管道中使用以下 NIM 微服务。您可以灵活地将管道中的组件更改为适用于不同模型的其他 NIM 微服务：
Llama3 70B Instruct (meta/llama3-70b-instruct)
：一种大语言模型，可根据检索到的文本生成用户问题的答案。
NV-EmbedQA-e5-v5
(nvidia/nv-embedqa-e5-v5)：嵌入来自 IFU 的文本块和用户查询的嵌入模型。
NV-RerankQA-Mistral-4b-v3 (nvidia/nv-rerankqa/mistral-4b-v3)
：一种重新排序模型，用于对检索到的文本块进行重新排序，以便由 LLM 执行文本生成步骤。
RIVA ASR：自动语音识别模型
，可将用户的语音查询转录为模型的文本。
RIVA TTS：输出 LLM 响应音频的文本转语音模型。
RAG 有两个步骤：文档摄取，然后检索和生成答案。可以在图 2 的参考架构图中找到这些步骤和相关的 NIM 微服务。
图 2、参考架构显示文档提取和检索
使用 NVIDIA NIM
您可以访问 NVIDIA 的 API Catalog（
build.nvidia.com
），注册免费的 API 积分，或在自己的计算基础设施上部署，以访问 NIM 微服务。
在本教程中，我们使用 API Catalog 端点。有关使用 NIM 微服务、查找 API 密钥以及其他预备知识的更多信息，请访问
GitHub
。
按照以下步骤构建具有可选语音的 RAG 工作流，以便使用其 IFU 回答医疗设备问题。
构建和启动容器
查看我们为使用 NIM 微服务和向量数据库启动容器而创建的 Docker Compose 文件。您可以在 GitHub 上获取
详细说明
和代码。
提取设备手册
导航浏览器，在“Knowledge Base”（知识库）选项卡中上传您的 IFU，如图 3 所示。
图 3、医疗设备训练助手的文档提取页面
检索并生成答案导航
至“Converse”选项卡，开始与 IFU 对话（图 1）。请务必单击“Use Knowledge Base”，将 IFU 用作知识资源。要使用语音进行交谈，请单击文本输入区域旁边的麦克风，RIVA ASR 模型将转录您的问 题。要接收语音输出，请单击“Enable TTS output”。更多信息关于使用和故障排除的 UI 是在 Github 文档。
在自定义数据集上进行评估使用自定义问题数据集和自动化 RAGAS 指标评估 RAG 工作流的性能。RAGAS 指标评估检索器和生成器的性能，是以自动化方式评估 RAG 工作流的常用方法。有关如何使用评估脚本的说明，请参阅
GitHub
。
开始使用
要开始使用此工作流，请访问
GenerativeAIExamples GitHub 资源库
，其中包含本教程中使用的所有代码以及大量文档。
有关 NIM 微服务的更多信息，您可以参阅官方
NIM 文档
了解详情，并在我们的 NVIDIA Developer
NIM 论坛
上提出问题。