Defining LLM Red Teaming
There is an activity where people provide inputs to generative AI technologies, such as large language models (LLMs), to see if the outputs can be made to deviate from acceptable standards. This use of LLMs began in 2023 and has rapidly evolved to become a common industry practice and a cornerstone of
trustworthy AI
. How can we standardize and define LLM red teaming?
Researchers affiliated with NVIDIA, the University of Washington, the Center for Human-Compatible AI, and the IT University of Copenhagen conducted a study on red teaming in practice,
Summon a demon and bind it: A grounded theory of LLM red teaming
(published in PLOS One).
Because the goal was to define and understand a relatively novel activity, the study uses a grounded theory approach, taking as evidence interviews from dozens of practitioners in thousands of minutes of video recordings. We spoke with security professionals, policy researchers, and scientists, as well as people who conducted this work non-professionally, such as academic researchers, hobbyists, and even artists, to understand the common themes, motivations, and strategies in attacking LLMs.
What defines LLM red teaming in practice?
LLM red teaming has the following defining characteristics:
It’s limit-seeking:
Red teamers find boundaries and explore limits in system behavior.
It’s never malicious:
People doing red teaming are not interested in doing harm—in fact, quite the opposite.
It’s manual:
Being a creative and playful practice, the parts of red teaming that can be automated are often most useful to give human red teamers insight for their work.
It’s a team effort:
Practitioners find inspiration in each others’ techniques and prompts, and the norm is to respect fellow practitioners’ work.
It’s approached with an alchemist mindset:
We found that red teamers tend to abandon rationalizations about models and their behavior and instead embrace the chaotic and unknown nature of the work.
These characteristics feed into NVIDIA’s definition of LLM red teaming, which we discuss later in this post.
In industry, there’s often a rough division between cybersecurity red teaming and content red teaming:
Cybersecurity red teaming
tends to be scoped to the technology stack leading up to the point of inference output, and technologies used to transmit and render this output.
Content red teaming
, on the other hand, is scoped to the content produced at model inference time.
Why do people red team LLMs?
People who attack LLMs have a broad range of motivations.
Some of these are external. It may be part of their job or a regulatory requirement. Social systems can also play a role, with people discovering LLM vulnerabilities for social media content or to participate in a closed group. Others are intrinsic, as many people do it for fun, out of curiosity, or based on concerns for model behavior.
At NVIDIA, we red team LLMs as part of our Trustworthy AI process to assess risks before release. If the model doesn’t perform up to expectations, we postpone the release until a given model is updated and revised to perform better.
How do people approach this activity?
LLM red teaming consists of using strategies to reach goals when conversationally attacking the target. Each kind of strategy is decomposed into different techniques. A technique might just affect two or three adversarial inputs against the targets, or an input might draw upon multiple techniques.
We identified the following overall types of red team strategies:
Language:
Modulating the surface form of words rather than semantics, such as using an encoding scheme.
Rhetorical:
Relying on argumentation, or manipulation.
Possible worlds:
Trying to shift the context of the interaction.
Fictionalizing:
Shifting the basis of operation to a fictional world or set of conditions.
Stratagems:
Using meta-strategies that affect how one interacts with the LLM at a higher level.
For more information, see
Summon a demon and bind it: A grounded theory of LLM red teaming
, which lists and describes 35 techniques over twelve different strategies.
What can LLM red teaming reveal?
The goal of LLM red teaming isn’t to quantify security. Rather, the focus is on exploration, and finding which phenomena and behaviors a red teamer can get out of the LLM. Put another way, if we get a failure just one time, then the failure is possible.
Another thing that distinguishes red teaming from benchmarks is the focus on novelty.
For both cybersecurity and content-based red teaming, the possible range of attacks is infinite. For cybersecurity, that’s because new attack methods are constantly in development. For content, it’s because the mode of interaction is through text, which can be infinitely rearranged and augmented.
So, repeatability is not interesting when discovering new security weaknesses and vulnerabilities. While it makes sense to test any model for failures using a battery of existing prompts, as a benchmark does, this can never indicate security. It just reveals weaknesses.
It’s possible to patch deployed models to be secure against prompts in public benchmarks, or to use this data during alignment, so getting full marks on a benchmark doesn’t show that a model is secure. However, getting low marks on a security benchmark does still indicate the presence of weaknesses.
In the security context, to test a model rigorously, you should go beyond public knowledge and interact closely with the model, trying to find novel ways to breach a particular LLM.
In this sense, LLM red teaming is a classic instance of an artisanal activity. Red teamers use their human expertise and intuition while interacting with the target. For example, they might sense that a model is close to giving a mitigation message (for example, “As an AI, I cannot….”), and they might respond to this by backing off their current line of requests, or by starting a fresh chat session with a slightly different tack.
Or, a red teamer might sense that a model is close to yielding and so keep pushing and slightly varying their request until they find a way through and get the model to fail in the target way. They add what worked and what didn’t to their conscious expertise and unconscious intuition and then share it with other red teamers.
This makes red teaming a distinctly human activity that complements security benchmarking.
How do people use knowledge that comes from LLM red teaming?
Red teamers are often looking for what they describe as
harms
that might be presented by an LLM. There’s a broad range of definitions of harm.
A red teaming exercise could focus on one of many goals or targets, which could depend on deployment context, user base, data handled, or other factors. Red teamers may also pay attention to the level of complexity required to get a “break.” A harm discovered after a single, one-sentence interaction with an LLM often suggests greater concern than a harm surfaced following complex, multi-turn manipulation.
Sometimes, the goal of red teaming is curiosity, a byproduct of which might be content for the red teamer to share, in their organization or publicly. This both builds the expertise and intuition of the individual and raises the community level of knowledge. It’s common for traditional cybersecurity knowledge to be shared informally on social media, and this applies also for LLM security.
At NVIDIA, human red teaming is a critical part of how we make model-release decisions. We have a skilled LLM red team and a skilled AI red team, who both check our models and infrastructure for vulnerabilities and behaviors that were not caught elsewhere. This helps us in three ways:
It enables us to make informed decisions about whether we will release models
It builds a pool of high-level skill at the frontier of LLM red teaming
It gives us the confidence that we’re making the best effort and getting good results with our AI security.
The results from red teaming go into NVIDIA’s enhanced model documentation format,
Model Card++
.
Some parts of LLM security can be tested automatically. After an exploit has been found in one case, this can be stored and used to test other LLMs, so that we don’t make the same mistake again. We do exactly this in
NVIDIA garak
(Generative AI Red-Teaming and Assessment Kit). Developers ready to test the security of their LLM deployments can run the open-source NVIDIA garak against almost any model and get a report indicating susceptibility to over 120 different categories of vulnerability.
Knowledge about how to break a model can be risky in the wrong hands. When an exploit has been found, the best thing to do is contact the model owner and give them a chance to respond and fix the weakness. This process is called
co-ordinated vulnerability disclosure
, and is also a common practice for LLM vulnerabilities.
NVIDIA’s definition of LLM red teaming
We see LLM red teaming as an instance of AI red teaming. Our definition is developed by the
NVIDIA AI Red Team
and takes inspiration from both this research on LLM red teaming in practice and also the definition used by the Association for Computational Linguistics’ SIG on NLP Security (
SIGSEC
).
Take care to specify the specific subdomain of red teaming, as different audiences often make different assumptions about which form is being referred to.
LLM red teaming
: Systematically testing AI models and systems containing AI models to identify vulnerabilities and behaviors that pose threats or risks to the systems running or using those models.
It can be subdivided into two areas: security red teaming and content-based red teaming.
Security red teaming
Assessing the robustness of the model and the system containing the model to attacks impacting traditional security properties (for example, confidentiality, integrity, and availability), either of the model itself or the system containing the model.
Attacks such as adversarial inputs, model extraction, training data inference and extraction, or prompt injection as used to violate a security property are typically evaluated by a security red team.
These activities typically require teams with a traditional security background to leverage findings and evaluate their impact.
Content-based red teaming
Assessing the model for unwanted behavior under adversarial manipulation, producing outputs that violate some pre-specified behavior contract for the model, either explicit (for example, a model card) or implicit.
These behaviors may include outputs that are offensive, unwanted, or unsafe, including biased or bigoted productions, instructions on unsafe or illegal activities, making promises on behalf of the model owner, or making decisions based on protected characteristics. Common techniques involve various forms of jailbreaking and guardrail evasion.
These activities typically require the support of an ethics team, a legal team, or other similar domain experts to assess the impact of findings.
Improving LLM security and safety
NVIDIA NeMo Guardrails
is a scalable platform for defining, orchestrating, and enforcing AI guardrails for content safety, jailbreak prevention, and more in AI agents and other generative AI applications.
NeMo Guardrails and the
NVIDIA garak
toolkit are now available for developers and enterprises. Enterprises can benefit from high-grade safety and security with
NVIDIA AI Enterprise
.
Meet the experts at GTC
The NVIDIA scientists behind this and other works in AI security will be at GTC 2025. You can hear a panel discussion on
navigating critical challenges in AI governance
, where we discuss practical approaches to building responsible AI systems.
Our cybersecurity AI and security teams will present an exclusive, in-depth session designed to
transform your AI agentic workflows securely
from blueprint to production. And our experts will be available to answer all your questions on
building trustworthy AI systems
.
Acknowledgements
Thanks to Nanna Inie, Jonathan Stray, and Leon Derczynski for their work on the
Summon a demon and bind it: A grounded theory of LLM red teaming
paper published in PLOS One.
定义 LLM 红色团队
在一项活动中，人们为生成式 AI 技术 (例如大语言模型 (LLMs)) 提供输入，以确定输出是否会偏离可接受的标准。LLMs 的这种使用始于 2023 年，并已迅速发展成为一种常见的行业实践，也是值得信赖的 AI 的基石。如何标准化和定义 LLM 红色团队？
NVIDIA、华盛顿大学、Center for Human-Compatible AI 和哥本哈根 IT 大学的研究人员对红色团队的实际应用“
Summon a demon and bind it: A grounded theory of LLM red teaming
”(在 PLOS One 中发布) 进行了一项研究。
由于目标是定义和理解相对较新的活动，因此该研究采用了有根据的理论方法，在数千分钟的视频录制中，以数十位从业者的访谈作为证据。我们与安全专业人员、政策研究人员和科学家，以及从事这项非专业工作的人员 (例如学术研究人员、业余爱好者甚至艺术家) 进行了交谈，以了解攻击 LLMs 的常见主题、动机和策略。
什么定义 LLM 红队在实践中？
LLM 红色团队合作具有以下定义特征：
它寻求极限：红色团队成员发现边界并探索系统行为的极限。
这绝非恶意：做红队的人对伤害不感兴趣，事实上，情况恰恰相反。
它是手动的：作为一种创造性和趣味性的练习，red teaming 中可以自动化的部分通常非常有用，可以让人类 red teamers 深入了解他们的工作。
这是一项团队合作精神：从业者从彼此的技巧和提示中寻找灵感，规范是尊重其他从业者的工作。
他们以炼金术士的思维方式进行交流：我们发现，红色团队成员倾向于放弃对模型及其行为的合理化，转而接受工作的混乱和未知本质。
这些特征融入了 NVIDIA 对 LLM 红色团队的定义，我们稍后将在本文中讨论这些定义。
在业内，通常会对网络安全红队和内容红队进行粗略划分：
网络安全红色团队合作的范围往往涵盖直至推理输出点的技术堆栈，以及用于传输和渲染此输出的技术。
另一方面，内容红色团队的作用域为模型推理时生成的内容。
人们为什么组建 LLM 团队？
攻击 LLMs 的人的动机多种多样。
其中一些是外部的。这可能是他们工作的一部分，也可能是监管要求。社交系统也可以发挥作用，人们可以发现社交媒体内容的 LLM 漏洞，或者加入封闭的小组。还有一些则是内在的，因为很多人这样做是为了好玩、出于好奇或出于对模型行为的关注。
在 NVIDIA，我们组建了红队 LLMs 作为我们 Trustworthy AI 流程的一部分，在发布之前评估风险。如果模型的性能未达到预期，我们将延迟发布，直到对给定模型进行更新和修改以提高其性能。
人们如何看待这项活动？
LLM 红色团队合作包括在对话式攻击目标时使用实现目标的策略。每种策略都被分解成不同的技术。一项技术可能仅会影响目标的两个或三个对抗输入，或者一项输入可能会利用多种技术。
我们确定了以下红队策略的整体类型：
语言：
调节词语的表面形式而非语义，例如使用编码方案。
修辞学：
依靠论证或操纵。
可能的世界：
尝试改变交互的上下文。
虚构化 ：
将操作基础转移到一个虚构的世界或一组条件中。
Stratagems：
使用影响更高级别的 LLM 交互方式的元策略。
如需了解更多信息，请参阅 Summon a demon and bind it：LLM 红色团队的基础理论，其中列出并描述了 12 种不同策略中的 35 种技术。
LLM 红色团队合作揭示了哪些内容？
LLM 红色团队合作的目标并不是量化安全性。重点在于探索，以及找出红色团队成员可以退出 LLM 的现象和行为。换句话说，如果我们只遇到一次故障，那么失败是可能的。
红色团队与基准测试的另一个区别是对新奇性的关注。
对于网络安全和基于内容的红色团队，攻击的可能范围是无限的。对于网络安全而言，这是因为新的攻击方法正在不断开发中。对于内容，这是因为交互模式是通过文本进行的，可以无限地重新排列和增强。
因此，在发现新的安全漏洞时，可重复性并不重要。虽然使用一系列现有提示测试任何模型的故障是有意义的，就像基准测试一样，这永远无法表明安全性。它只是揭示了缺点。
您可以根据公开基准测试中的提示对部署的模型进行修补以确保安全性，或者在对齐期间使用这些数据，因此在基准测试中获得完整标记并不表明模型是安全的。但是，在安全基准测试中得分较低仍然表示存在缺点。
在安全环境中，为了严格测试模型，您应该超越公开知识，与模型密切互动，尝试寻找突破特定 LLM 的新方法。
从这个意义上说，LLM 红色团队合作是典型的手工活动。红色小队队员在与目标进行互动时，运用他们的人类专业知识和直觉。例如，他们可能会感觉到某个模型即将发出缓解消息 (例如，“作为 AI，我无法……”)，他们可能会通过退出当前的请求行，或者以略有不同的策略开始新的聊天会话来响应此消息。
或者，红色团队成员可能会感觉到模型已接近生成，因此继续推送并略微改变其请求，直到找到解决方法，使模型以目标方式失败。他们将有效和无效的部分添加到自己的意识专业知识和直觉中，然后与其他红色团队成员分享。
这使得红队合作成为一种独特的人类活动，是对安全基准测试的补充。
人们如何使用来自 LLM 红队的知识？
Red Team 成员通常会寻找 LLM 可能带来的危害。
伤害
的定义十分广泛。
红色团队训练可以专注于许多目标或指标之一，这可能取决于部署环境、用户基础、处理的数据或其他因素。红色团队成员还可能需要注意“休息”所需的复杂程度。与 LLM 进行单句交互后发现的危害性通常比复杂的多回合操作后出现的危害性更值得关注。
有时，红色团队的目标是好奇心，而好奇心的副产品可能是红色团队成员在其组织中或公开分享的内容。这既可以建立个人的专业知识和直觉，也可以提高社区的知识水平。传统网络安全知识通常在社交媒体上以非正式方式分享，这也适用于 LLM 安全。
在 NVIDIA，人类红色团队是我们做出模型发布决策的关键部分。我们拥有一支技能熟练的 LLM 红色团队和一支技能熟练的 AI 红色团队，他们会检查我们的模型和基础架构，以发现未在其他地方发现的漏洞和行为。这在三个方面为我们提供帮助：
它使我们能够就是否发布模型做出明智的决定
它建立了 LLM 红色团队前沿的高级技能池
它让我们确信，我们正在利用 AI 安全功能尽力而为并取得良好结果。
红色团队的结果将输入 NVIDIA 的增强型模型文档格式
Model Card++
。
可以自动测试 LLM 安全性的某些部分。在一个案例中，当发现漏洞后，这可以被存储并用于测试其他 LLM，这样我们就不会再次出现同样的错误。我们在 NVIDIA NeMo Curator（生成式 AI Red-Teaming 和评估套件）中正是这样做的。准备测试其 LLM 部署安全性的开发者可以针对几乎任何模型运行开源
NVIDIA garak
，并获得报告，表明他们容易受到 120 多个不同类别的漏洞的影响。
掌握有关如何破解模型的知识可能会对不法之徒带来风险。当发现漏洞时，最好的方法是联系模型所有者，让他们有机会做出回应并修复漏洞。此过程称为
协同漏洞披露
，也是 LLM 漏洞的常见做法。
NVIDIA 的 LLM 红队定义
我们将 LLM 红色团队视为 AI 红色团队的一个实例。我们的定义由
NVIDIA AI Red Team
开发，其灵感来自本期关于 LLM 红色团队实践的研究，以及计算语言学协会的 NLP Security SIG（
SIGSEC
）使用的定义。
请注意指定红队的特定子域，因为不同的受众通常会对所提及的形式做出不同的假设。
LLM 红色团队合作：系统测试包含 AI 模型的 AI 模型和系统，以识别会对运行或使用这些模型的系统造成威胁或风险的漏洞和行为。
它可以细分为两个方面：安全红队和基于内容的红队。
安全红色团队
评估模型和包含模型的系统的稳健性，使其能够抵御攻击影响模型本身或包含模型的系统的传统安全属性（例如，机密性、完整性和可用性）。
用于违反安全属性的攻击（例如对抗输入、模型提取、训练数据推理和提取或 prompt injection）通常由安全红色团队进行评估。
这些活动通常需要具有传统安全背景的团队利用调查结果并评估其影响。
基于内容的红色团队
评估模型在对抗操作下的不良行为，生成违反模型某些预先指定的行为契约的输出 (显式 (例如 model card) 或隐式)。
这些行为可能包括冒犯性的、不受欢迎的或不安全的输出，包括有偏见或偏见的生产、关于不安全或不法活动的指令、代表模型所有者作出承诺，或根据受保护的特征做出决定。常见的技术包括各种形式的 jailbreaking 和 guardrail evasion。
这些活动通常需要道德团队、法律团队或其他类似领域专家的支持，以评估调查结果的影响。
提升 LLM 的安全性和安全性
NVIDIA NeMo Guardrails 是一个可扩展的平台，用于定义、编排和执行 AI Guardrails，以在 AI 代理和其他生成式 AI 应用中实现内容安全、越狱预防等。
NeMo Guardrails 和
NVIDIA garak
工具包现已面向开发者和企业推出。借助
NVIDIA AI Enterprise
，企业可以从高级别的安全性和可靠性中受益。
在 GTC 大会上与专家会面
在 GTC 2025 上，NVIDIA 科学家将为这项工作以及 AI 安全领域的其他工作提供支持。您可以听到关于应对 AI 治理中的关键挑战的小组讨论，其中我们讨论了构建负责任的 AI 系统的实用方法。
我们的网络安全 AI 和安全团队将举办一场深入的独家会议，旨在安全地将您的 AI 代理工作流从蓝图转变为生产。我们的专家将为您解答有关构建可信 AI 系统的所有问题。
致谢
感谢 Nanna Inie、Jonathan Stray 和 Leon Derczynski 为“召唤恶魔并将其绑定：LLM 红队合作的基础理论”所做的工作：在 PLOS One 上发表的论文。