Latest Multimodal Addition to Microsoft Phi SLMs Trained on NVIDIA GPUs
Large language models (LLMs) have permeated every industry and changed the potential of technology. However, due to their massive size they are not practical for the current resource constraints that many companies have.
The rise of
small language models
(SLMs) bridge quality and cost by creating models with a smaller resource footprint. SLMs are a subset of language models that tend to focus on specific domains and are built with simpler neural architectures. As models grow to mimic how humans perceive the world around them, models must rise to accept multiple forms of multimodal data.
Microsoft announces
the new generation of open SLMs
to the Phi family with two new additions:
Phi-4-mini
Phi-4-multimodal
Phi-4-multimodal is the first multimodal model to join the family that accepts text, audio, and image data inputs.
These models are small enough for on-device deployment. This release builds on top of the December 2024 research-only release of the
Phi-4 14B parameter SLM
and enables commercial use for the two new smaller models.
The new models are available on the
Azure AI Foundry
, Microsoft’s Cloud AI platform for design, customize, and manage AI applications and agents.
You can test out each member of the Phi family through the
NVIDIA API Catalog
, which is the first sandbox environment to support each modality and tool-calling for
Phi-4-multimodal
. Use the preview NIM microservice to integrate the model into your applications today.
Why invest in SLMs?
SLMs enable generative AI capabilities in memory and compute constrained environments. For example, SLMs can be deployed directly on smartphones and several consumer-grade devices. On-device deployment can facilitate privacy and compliance for use cases that must adhere to regulatory requirements.
Other benefits of SLMs include lower latency due to inherently faster inference compared to an LLM of similar quality. SLMs do tend to perform better on specialized tasks correlated to their training data. However, to supplement generalization and adaptability to different tasks, you can use retrieval-augmented generation (RAG) or native-function calling to build performant agentic systems.
Phi-4-multimodal
Phi-4-multimodal is with 5.6B parameters and accepts audio, image, and text reasoning. This enables it to support use cases such as automated speech recognition (ASR), multi-modal summarization, translation, OCR, and visual reasoning. This model was trained on 512 NVIDIA A100-80GB GPUs over 21 days.
Figure 1 shows how you can preview your image data and ask Phi-4-multimodal visual QA in the NVIDIA API Catalog. You can also see how to adjust parameters such as token limits, temperature, and sampling values. You can generate sample code in Python, JavaScript, and Bash to help you integrate the model more easily into your applications.
Figure 1. Visual QA demo in NVIDIA API Catalog
You can also demo tool calling with a set of prebuilt agents. Figure 2 shows a tool that retrieves live weather data.
Figure 2. Tool-calling demo in NVIDIA API Catalog
Phi-4-mini
Phi-4-mini is a text-only, dense, decoder-only Transformer model with 3.8B parameters that is optimized for chat. It includes a long-form context window of 128K tokens. This model was trained on 1024 NVIDIA A100 80GB GPUs over 14 days.
For both models, the training data is intentionally focused on high quality educational data and code which results in a
textbook-like quality
to the models. Text, speech, and vision benchmark data can be found in the model cards.
Advancing community models
NVIDIA is an active contributor to the open-source ecosystem and has released several hundred projects under open-source licenses. NVIDIA is committed to optimizing community software and open models such as Phi which promotes AI transparency and lets users broadly share work in AI safety and resilience.
Using the
NVIDIA NeMo platform
, these open models can be customized on proprietary data to be highly tuned and efficient for diverse AI workflows across any industry.
NVIDIA and Microsoft have a long standing partnership which includes several collaborations driving innovation on GPUs on Azure, integrations and optimizations for PC developers using NVIDIA RTX GPUs, and many more, including research spanning generative AI to healthcare and life sciences.
Get started today
Bring your data and try out Phi-4 on the NVIDIA-accelerated platform at
build.nvidia.com/microsoft
.
On the first multi-modal sandbox for Phi-4-multimodal, you can try out text, image, and audio as well as sample tool calling to see how this model will work for you in production.
在 NVIDIA GPU 上训练的 Microsoft Phi SLM 的多模态最新进展
大语言模型（LLMs）已渗透到各行各业，并改变了技术潜力。但是，由于规模庞大，它们对于许多公司目前面临的资源限制来说并不切实际。
小语言模型
（SLMs）的兴起通过创建资源占用更小的模型，将质量和成本联系起来。SLMs 是语言模型的一个子集，这些模型倾向于专注于特定领域，并使用更简单的神经架构构建。随着模型的发展模仿人类感知周围环境的方式，模型必须接受多种形式的多模态数据。
Microsoft 宣布在 Phi 系列中
推出新一代开放式 SLM
，并新增两项功能：
Phi-4-mini
Phi-4-multimodal
Phi-4-multimodal 是第一个加入该系列的多模态模型，接受文本、音频和图像数据输入。
这些模型足够小，可以在设备上部署。此版本基于 2024 年 12 月发布的
Phi-4 14B 参数 SLM
的研究版本构建而成，可用于两个新的较小模型的商业用途。
这些新模型可在 Microsoft 的云 AI 平台
Azure AI Foundry
上使用，用于设计、定制和管理 AI 应用和代理。
您可以通过
NVIDIA API Catalog
测试 Phi 系列的每个成员，这是第一个支持
Phi-4 多模态
的每种模式和工具调用的沙盒环境。立即使用预览 NIM 微服务将模型集成到您的应用中。
为何投资 SLM？
SLMs 可在内存和计算受限环境中实现生成式 AI 功能。例如，SLMs 可以直接部署在智能手机和多台消费级设备上。对于必须遵守监管要求的用例，设备端部署可以促进隐私和合规性。
SLM 的其他优势包括降低延迟，因为与质量相似的 LLM 相比，其本身的推理速度更快。SLM 在处理与其训练数据相关的专业任务时往往表现得更好。但是，为了补充对不同任务的泛化和适应性，您可以使用检索增强生成（RAG）或原生函数调用来构建高性能代理系统。
Phi-4-multimodal
Phi-4-multimodal 具有 5.6B 个参数，接受音频、图像和文本推理。这使其能够支持自动语音识别 (ASR)、多模态摘要、翻译、OCR 和视觉推理等用例。该模型在 512 个 NVIDIA A100-80GB GPUs 上进行了为期 21 天的训练。
事实证明，该模型在 ASR 方面表现出色，因为它在
Huggingface OpenASR 排行榜上排名第一
，单词错误率为 6.14%。
词错误率 (WER)
是量化语音识别性能的常用计算方法。WER 计算不正确转录的单词 (替换、插入和删除) 与正确文本相比所占的百分比。
图 1 展示了如何在 NVIDIA API Catalog 中预览图像数据并询问 Phi-4 多模态视觉问答。您还可以了解如何调整参数，例如令牌限制、温度和采样值。您可以使用 Python、JavaScript 和 Bash 生成示例代码，以帮助您更轻松地将模型集成到应用中。
图 1、NVIDIA API Catalog 中的可视化问答演示
您还可以使用一组预构建代理演示工具调用。图 2 显示了用于检索实时天气数据的工具。
图 2、NVIDIA API Catalog 中的工具调用演示
Phi-4-mini
Phi-4-mini 是一个仅文本、密集、仅解码器的 Transformer 模型，具有 3.8B 个参数，并针对聊天进行了优化。它包含一个包含 128K 个令牌的长形式上下文窗口。该模型在 1024 个 NVIDIA A100 80GB GPUs 上进行了为期 14 天的训练。
对于这两个模型，训练数据有意地集中在高质量的教育数据和代码上，从而使模型获得类似于教科书的质量。您可以在模型卡中找到文本、语音和视觉基准测试数据。
推进社区模式
NVIDIA 是开源生态系统的积极贡献者，已根据开源许可发布了数百个项目。NVIDIA 致力于优化社区软件和 open-source licenses 中的项目，如 Phi，它促进了 AI 透明度，并让用户广泛分享在 AI 安全性和弹性方面的工作。
借助 NVIDIA NeMo 平台，这些开放模型可以根据专有数据进行定制，以便针对各行各业的各种 AI 工作流进行高度调整并提高效率。
NVIDIA 和 Microsoft 有着长期的合作伙伴关系，其中包括推动 Azure 上 GPU 创新的多项合作、为使用 NVIDIA RTX GPU 的 PC 开发者提供的集成和优化，等等，包括从生成式 AI 到医疗健康和生命科学的研究。
立即开始使用
请访问
build.nvidia.com/microsoft
，带上您的数据并在 NVIDIA 加速平台上试用 Phi-4。
在 Phi-4 多模态的第一个多模态沙盒中，您可以尝试使用文本、图像、音频以及示例工具调用，以了解此模型在生产环境中的工作原理。