Vision Language Model Prompt Engineering Guide for Image and Video Understanding
Vision language models
(VLMs) are evolving at a breakneck speed. In 2020, the first VLMs revolutionized the
generative AI
landscape by bringing visual understanding to
large language models
(LLMs) through the use of a
vision encoder
. These initial VLMs were limited in their abilities, only able to understand text and single image inputs.
Fast-forward a few years and VLMs are now capable of understanding multi-image and video inputs to perform advanced vision-language tasks such as visual question-answering (VQA), captioning, search, and summarization.
Figure 1. Suite of model customization tools in NVIDIA NeMo
VLM accuracy can be improved for specific use cases by tuning the prompts and model weights. Fine-tuning the model weights can be done with efficient techniques such as PEFT but still requires sufficient data and compute.
However,
prompt engineering
and in context learning can also be used to improve the output quality and is as simple as adjusting the text input to the model at runtime.
This post walks you through the evolution of VLMs and how to effectively prompt a VLM for single image, multi-image, and video understanding use cases. For more information about VLMs and visual AI agents, register for the upcoming
Vision for All: Unlocking Video Analytics with AI Agents
webinar.
Single-image understanding
From a single image, a VLM is capable of identifying, describing, classifying, and reasoning over what is in the image. Some VLMs have advanced capabilities such as providing bounding box coordinates for certain objects, and translating text in the image. This works best for use cases where an image needs to be classified, captioned or described in text.
Single-image capability can also be used to detect basic events occurring in a livestream. This can be accomplished by sampling one frame at a time from a livestream and providing it to the VLM along with a prompt describing what to look for. This works especially well when the types of events to detect are large, clearly visible, and slow-moving.
The VLM can also be prompted to output in a structured format such as JSON so that the response can easily be parsed and sent to a database or a notification service.
For example, we provided Llama-3.2-90b-vision-instruct with the following prompt and image (Figure 2).
Figure 2. Input image to the VLM for question and answering
Prompt:
Is there a fire truck? Is there a fire? Are there firefighters? Output the answer to each question in JSON format.
VLM response:
[ { "question": "Is there a fire truck?", "answer": "True" }, 
  { "question": "Is there a fire?", "answer": "True" }, 
  { "question": "Are there firefighters?", "answer": "True" } 
]
For each of the questions, the VLM was able to accurately respond and output it in a structured format to be used for downstream tasks.
While this method enables the VLM to operate on video input by sampling one frame at a time, the VLM itself does not understand the full context of the video, limiting it to simple use cases. Sampling one frame at a time does not give it the temporal understanding needed to detect actions or trends that occur over time. This can be improved with multi-image and video VLMs.
Multi-image understanding
Many VLMs can accept multiple images as input allowing the model to compare, contrast, and learn from input images.
VLMs generally have good zero-shot capability across a wide range of use cases. However, when trying to obtain more precise answers from a VLM on domain-specific use cases, VLMs can struggle because they lack the full context required to accurately respond to the user prompts.
To better understand this, we used a VLM to estimate the stock level of products. This is an important retail use case as it could autonomously monitor and alert employees when products are running low in a store to avoid lost revenues from stock-outs.
We provided a single image of a table (Figure 3) that should be fully stocked with several snack products to
Cosmos Nemotron 34B
and asked it to estimate the stock level.
Figure 3. Single-image VLM input for retail stock level detection
Prompt:
Estimate the stock level of the snack table on a scale of 0–100.
VLM response:
The snack table appears to be well-stocked, with a variety of snacks and treats. The shelves are filled with an assortment of items, including chips, cookies, and other snacks. The table is also neatly organized, with each item placed in a designated spot. Based on the visual, I would rate the stock level at around 80-90, indicating that the table is well-stocked and ready to cater to a crowd.
The model rates it as 80–90% stocked even though it’s quite clear to the human eye that the stock level is low. From the model’s perspective, it does not know what the table looked like beforehand when looking at a single image.
Using the Cosmos Nemotron 34B multi-image capability, we provided another image of the snack table at a 100% stock level (Figure 4), giving  the model extra context to produce a more accurate estimate.
Figure 4. In-context learning input to VLM for retail stock level detection
Prompt:
First compare and contrast the stock level of the two images. Then generate an estimate for each image of the stock level on a scale of 0–100%.
VLM response:
The first image shows a well-stocked display with a variety of snacks, while the second image shows a more sparse display with only a few items. Based on the visual comparison, I estimate that the first image has a stock level of around 80-90%, while the second image has a stock level of around 20-30%.
By providing the additional image as a reference, the VLM now estimates the stock level of the second image at 20–30%, which is significantly more accurate.
Multi-image input can also be used in more complex multimodal RAG pipelines where example images are dynamically added to the prompt based on the user’s input. Multi-image VLMs can also be used for basic video understanding by sampling a small set of frames from a video and providing it to the VLM.
However, many VLMs that accept multi-image input have limited context windows (10–20 frames) and may not have been trained on video data. To truly understand video input, we recommend using a VLM trained on video-specific datasets and a long context window.
Video understanding
The latest VLMs have full video understanding capabilities with large context lengths, frame compression and video training data. This enables VLMs to ingest longer videos and have improved temporal understanding: the ability to understand actions and trends over time.
The first example showed how to use a single frame from a livestream to detect a fire but that same method wouldn’t be able to tell you if the fire is growing or getting smaller because it lacks the temporal context.
A VLM with long context and video understanding processes many frames across time to understand if the fire is growing or not. Here are some examples from the Cosmos Nemotron model families.
Sequential visual understanding
VLMs with sequential understanding relies on processing frames in a sequence, enabling them to capture the progression of actions and events over time. Unlike multi-image models, which process a limited number of frames without explicit temporal connections, sequential video models incorporate temporal dynamics by analyzing frames in an orderly sequence.
This is helpful in situations where you want to capture the causality of events in a video, as it enables the model to track the progression of actions over time and establish cause-effect relationships.
The next few examples are based on the following video where a worker drops a box in a warehouse. We discuss a few prompts and responses.
Figure 5. Input warehouse video to the VLM for scene understanding
Prompt:
What happened in this video?
VLM response:
The video shows a warehouse worker in a yellow safety vest and hard hat performing various tasks. The worker is seen walking through the warehouse, carrying boxes, and interacting with the shelving units. The warehouse has green metal shelving with labels like "C", "D", "E", and "F" on the aisles.

The worker appears to be in the process of organizing or retrieving items from the shelves. At one point, they're seen bending down to pick up a box from the floor. The worker's movements suggest they're actively engaged in their duties, possibly sorting, stocking, or preparing items for shipment.

The warehouse environment is well-lit, with fluorescent lighting illuminating the space. The worker's safety gear, including the reflective vest and hard hat, indicates adherence to safety protocols in this industrial setting.

Overall, the video captures a typical day in the life of a warehouse worker, showcasing the physical nature of the job and the organized layout of the storage facility.
The model gave a detailed response with specifics about the environment such as aisle labels and lighting conditions but failed to correctly identify that the worker dropped a box. Now try to ask this question directly.
Prompt:
Did the worker drop any box?
VLM response:
Yes, the worker did drop a box. In the video, we can see a box lying on the floor in front of the worker. This suggests that the worker has accidentally dropped the box while handling it.
The VLM is now able to deduce that the worker might have dropped the box. This suggests that generic prompts might not capture all the nuances correctly unless you direct the VLM to look for something specific. Modify the prompt a little.
Prompt:
Elaborate on the visual and narrative elements of this warehouse video in detail. Highlight all anomalies or hazards if present.
VLM response:
The video captures a warehouse with tall metal shelving units labeled with letters 'C', 'D', 'E', and 'F'. The shelves are stocked with various items, including boxes and packages. The floor is concrete, and the lighting is artificial, typical of an indoor industrial setting. A worker wearing a high-visibility vest, a hard hat, and dark pants is seen walking down the aisle between the shelves. The worker is carrying a cardboard box and then drops the box on the floor.
The model is now capturing all the important details of the scene along with important events, such as workers dropping a box. The detailed response is useful if you want to perform Q&A with the video later. VLMs with video understanding are also able to address directional questions, as they can understand actions over time.
Prompt:
From which direction did the worker enter the scene?
VLM response:
The worker entered the scene from the right side.
However, these models often lack precise temporal localization, which can limit their ability to pinpoint when specific events occur.
Prompt:
When did the worker drop the box?
VLM response:
The worker dropped the box in the second image.
Temporal localization
VLMs incorporating
Localization Interpretable Temporal Attention (LITA)
or similar temporal localization techniques elevate video understanding by explicitly learning when and where critical events occur. The models achieve this by introducing temporal attention mechanisms that focus on key segments of a video, aligning visual and textual modalities more effectively over time. This helps answer the “when” questions in the videos.
Prompt:
When did the worker drop the box?
VLM response:
0:04-0:06 The worker dropped the box between 4s and 6s. During this time, the worker is seen dropping the box on the ground.
The optimal prompt format for a VLM depends on the model’s architecture and the nature of the caption pairs used during training. Different training datasets influence how a VLM interprets the prompt.
Conclusion
This post walked through how VLMs have evolved from supporting only single-image input to being capable of complex temporal reasoning on long video inputs. To get started with VLMs, visit
build.nvidia.com
and try out some of the prompts shown in this post. For technical questions, see the
Visual AI Agent forum
.
VLMs can be used to build various visual AI agents. Explore one of the sample vision AI workflows on the
/NVIDIA/metropolis-nim-workflows
GitHub repo with no GPU required. To build a video analytics AI agent, try the
AI blueprint for video search and summarization
.
For more information about VLMs and visual AI agents, register for the upcoming
Vision for All: Unlocking Video Analytics with AI Agents
webinar.
For more information about LLM prompting, see
An Introduction to Large Language Models: Prompt Engineering and P-Tuning
.
图像和视频理解的视觉语言模型提示工程实践指南
视觉语言模型 (VLMs) 正在以极快的速度发展。2020 年，首批 VLMs 通过使用视觉编码器将视觉理解引入大语言模型 (LLMs) ，彻底改变了生成式 AI 格局。这些初始 VLMs 能力有限，只能理解文本和单张图像输入。
几年后，VLM 现在能够理解多图像和视频输入，以执行高级视觉语言任务，例如视觉问答 (VQA)、字幕、搜索和摘要。
图 1、NVIDIA NeMo 中的模型自定义工具套件
通过调整提示和模型权重，可以提高特定用例的 VLM 准确性。可以使用 PEFT 等高效技术微调模型权重，但仍需要足够的数据和计算。
但是，
prompt engineering
和上下文学习也可用于提高输出质量，并且与在运行时调整模型的文本输入一样简单。
本文将为您介绍 VLM 的演变，以及如何有效地提示 VLM 以用于了解单张图像、多张图像和视频的用例。如需了解有关 VLM 和视觉 AI 智能体的更多信息，请注册参加即将举办的
Vision for All：Unlocking Video Analytics with AI Agents 在线研讨会
。
单图像理解
从单张图像来看，VLM 能够识别、描述、分类和推理图像中的内容。一些 VLM 具有高级功能，例如为某些对象提供边界框坐标，以及翻译图像中的文本。这最适合需要对图像进行分类、添加字幕或以文本进行描述的用例。
单图像功能还可用于检测直播中发生的基本事件。这可以通过以下方式实现：从直播中一次采样一帧，并将其提供给 VLM，同时提供描述要查找的内容的提示。当要检测的事件类型较大、清晰可见且移动缓慢时，此方法的效果尤为出色。
系统还可以提示 VLM 以 JSON 等结构化格式输出，以便轻松解析响应并将其发送至数据库或通知服务。
例如，我们为 Llama-3.2-90b-vision-instruct 提供了以下提示和图像（图 2）。
图 2、将图像输入 VLM 以进行问答
提示：是否有消防车？是否有火灾？有消防人员吗？输出每个问题的答案，以 JSON 格式。
VLM 响应：
[ { "question": "Is there a fire truck?", "answer": "True" }, 
  { "question": "Is there a fire?", "answer": "True" }, 
  { "question": "Are there firefighters?", "answer": "True" } 
]
对于每个问题，VLM 能够准确回答并以结构化格式输出，以便用于下游任务。
虽然此方法允许 VLM 通过一次采样一帧来操作视频输入，但 VLM 本身无法理解视频的完整上下文，将其局限于简单的用例。一次对一帧进行采样无法为其提供检测随时间变化的动作或趋势所需的时间理解。这可以通过多图像和视频 VLM 得到改善。
多图像理解
许多 VLM 可以接受多张图像作为输入，允许模型比较、对比并从输入图像中学习。
VLM 通常在各种用例中都具有良好的零样本击能力。但是，在尝试从 VLM 获取有关特定领域用例的更精确答案时，VLM 可能会遇到困难，因为它们缺乏准确响应用户提示所需的完整上下文。
为了更好地理解这一点，我们使用 VLM 来估计产品的库存水平。这是一个重要的零售用例，因为它可以在商店中的产品运行不足时自动监控和提醒员工，以避免因缺货而损失的收入。
我们向
Cosmos Nemotron 34B
提供了一张表格 (图 3) 的单一图像，该表格应与多种零食产品一起充分库存，并要求其估算库存水平。
图 3. 用于零售库存水平检测的单图像 VLM 输入
提示：按 0 – 100 的比例估算零食表的库存量。
VLM 响应：
The snack table appears to be well-stocked, with a variety of snacks and treats. The shelves are filled with an assortment of items, including chips, cookies, and other snacks. The table is also neatly organized, with each item placed in a designated spot. Based on the visual, I would rate the stock level at around 80-90, indicating that the table is well-stocked and ready to cater to a crowd.
该模型将其评为 80–90% 的库存，即使肉眼很清楚库存水平很低。从模型的角度来看，它不知道在查看单个图像之前表格的样子。
通过使用 Cosmos Nemotron 34B 多图像功能，我们提供了另一张 100%库存水平的零食表图像 (图 4)，为模型提供了额外的背景信息，以便生成更准确的估计。
图 4. 为 VLM 提供情境学习输入，用于零售库存水平检测
提示：首先比较两张图像的库存水平。然后，按 0 – 100% 的比例对库存量的每张图像生成估计值。
VLM 响应：
The first image shows a well-stocked display with a variety of snacks, while the second image shows a more sparse display with only a few items. Based on the visual comparison, I estimate that the first image has a stock level of around 80-90%, while the second image has a stock level of around 20-30%.
现在，通过提供额外的图像作为参考，VLM 可将第二张图像的库存量估算为 20–30%，准确度明显更高。
多图像输入还可用于更复杂的多模态 RAG 工作流，其中会根据用户的输入将示例图像动态添加到提示中。多图像 VLM 还可以通过从视频中采样一小部分帧并将其提供给 VLM 来用于基本的视频理解。
但是，许多接受多图像输入的 VLM 的上下文窗口有限（10–20 帧），并且可能未基于视频数据进行训练。为了真正理解视频输入，我们建议使用基于视频特定数据集训练的 VLM 和长上下文窗口。
视频理解
最新的 VLM 具有完整的视频理解功能，具有大上下文长度、帧压缩和视频训练数据。这使得 VLM 能够提取更长的视频，并提高时间理解能力：能够理解一段时间内的行动和趋势。
第一个示例展示了如何使用直播中的单帧来检测火灾，但由于缺乏时间背景，相同的方法无法告诉您火灾是在增大还是变小。
具有长上下文和视频理解的 VLM 会随着时间的推移处理多个帧，以了解火势是否在增长。以下是 Cosmos Nemotron 模型系列的一些示例。
顺序视觉理解
具有顺序理解的 VLMs 依赖于按序列处理帧，使其能够捕捉动作和事件随时间推移的进展。与多图像模型相比，它们在处理有限数量的帧时没有显式时间连接，而顺序视频模型则通过按顺序分析帧来整合时间动力学。
这在您想要在视频中捕捉事件因果关系的情况下非常有用，因为它使模型能够跟踪动作在一段时间内的进展情况，并建立因果关系。
接下来的几个示例基于以下视频，展示了一名工人在仓库中丢弃箱子的过程。我们将讨论一些提示和响应。
图 5、将仓库视频输入到 VLM 以进行场景理解
提示：本视频中发生了什么？
VLM 响应：
The video shows a warehouse worker in a yellow safety vest and hard hat performing various tasks. The worker is seen walking through the warehouse, carrying boxes, and interacting with the shelving units. The warehouse has green metal shelving with labels like "C", "D", "E", and "F" on the aisles.

The worker appears to be in the process of organizing or retrieving items from the shelves. At one point, they're seen bending down to pick up a box from the floor. The worker's movements suggest they're actively engaged in their duties, possibly sorting, stocking, or preparing items for shipment.

The warehouse environment is well-lit, with fluorescent lighting illuminating the space. The worker's safety gear, including the reflective vest and hard hat, indicates adherence to safety protocols in this industrial setting.

Overall, the video captures a typical day in the life of a warehouse worker, showcasing the physical nature of the job and the organized layout of the storage facility.
该模型提供了详细的环境细节 (例如通道标签和照明条件)，但无法正确识别工人掉落的箱子。现在，尝试直接问这个问题。Prompt：工作人员是否丢弃了任何箱子？
VLM 响应：
Yes, the worker did drop a box. In the video, we can see a box lying on the floor in front of the worker. This suggests that the worker has accidentally dropped the box while handling it.
现在，VLM 可以推断出工作者可能丢弃了箱子。这表明，除非您指示 VLM 查找特定内容，否则通用提示可能无法正确捕捉所有细微差别。稍微修改一下提示。
提示：详细介绍此仓库视频的视觉和叙事元素。突出显示所有异常或危险（如果存在）。
VLM 响应：
The video captures a warehouse with tall metal shelving units labeled with letters 'C', 'D', 'E', and 'F'. The shelves are stocked with various items, including boxes and packages. The floor is concrete, and the lighting is artificial, typical of an indoor industrial setting. A worker wearing a high-visibility vest, a hard hat, and dark pants is seen walking down the aisle between the shelves. The worker is carrying a cardboard box and then drops the box on the floor.
现在，该模型可以捕捉场景的所有重要细节以及重要事件，例如工人丢箱。如果您想稍后在视频中执行 Q&A，详细的回答非常有用。具有视频理解功能的 VLM 还能够解决定向问题，因为它们可以理解随时间推移的动作。
提示：工作者是从哪个方向进入场景的？
VLM 响应：
The worker entered the scene from the right side.
但是，这些模型通常缺乏精确的时间定位，这可能会限制其确定特定事件何时发生的能力。
提示：工人是什么时候丢弃盒子的？
VLM 响应：
The worker dropped the box in the second image.
时间定位
结合
Localization Interpretable Temporal Attention (LITA)
或类似时间定位技术的 VLM 通过显式学习关键事件发生的时间和地点来提高视频理解能力。这些模型通过引入专注于视频关键片段的时间注意力机制，随着时间的推移更有效地调整视觉和文本模式来实现这一目标。这有助于回答视频中的“何时”问题。
提示：
工人是什么时候丢弃盒子的？
VLM 响应：
0:04-0:06 The worker dropped the box between 4s and 6s. During this time, the worker is seen dropping the box on the ground.
VLM 的最佳提示格式取决于模型的架构和训练期间使用的描述对的性质。不同的训练数据集会影响 VLM 解释提示的方式。
结束语
本文介绍了 VLM 如何从仅支持单图像输入发展为能够对长视频输入进行复杂的时间推理。要开始使用 VLM，请访问
build.nvidia.com
并尝试本博文中显示的一些提示。有关技术问题，请参阅
Visual AI Agent 论坛
。
VLM 可用于构建各种视觉 AI 智能体。无需 GPU，即可在
/NVIDIA/metropolis-nim-workflows
GitHub 存储库上探索视觉 AI 工作流示例之一。要构建视频分析 AI 智能体，请试用
NVIDIA AI Blueprint
中的视频搜索和摘要蓝图。
如需了解有关 VLM 和视觉 AI 智能体的更多信息，请注册参加即将举办的 Vision for All：Unlocking Video Analytics with AI Agents 网络研讨会。
有关 LLM 提示的更多信息，请参阅《
An Introduction to Large Language Models: Prompt Engineering and P-Tuning
》。