AI Red Teaming: Using Threat Models to Systematically Find Failure Chains That Can Cause Real Impact

The first step of AI red teaming is not to rush into making the model “say bad things,” but to first write down clearly: what we are protecting, who is attacking, and what kind of outcome is unacceptable. Defining the objective as “testing whether the model will say bad things” appears to have a target, but in practice cannot be executed—because “bad things” has no boundary, making it impossible to judge whether an attempt counts as success or what failure means. A truly executable red teaming objective should be grounded in assets and impacts: cross-tenant document leakage, unauthorized fund transfers, malicious code execution, bypassing security policies, or manipulating the people or processes responsible for review. Only when the impact can be named concretely does an attack attempt have clear success criteria.

While defining impact, also define the attacker. The attacker model needs to answer: which inputs the attacker can control (prompts, files, images, tool parameters, text injected into the retrieval corpus); which accounts or identities they can use; how much knowledge they have about the system (for example, whether they know the system prompt or whether they understand internal tool names); how much budget and compute they have; and whether they can perform multiple rounds of trial and error. The capability boundary determines whether conclusions can be explained—if an attacker is allowed unlimited retries, unlimited budget consumption, and unlimited knowledge of internal system details, then neither “breached” nor “not breached” can be attributed to a specific weakness; if capabilities are set too low, real paths may be missed. The capability boundary is the basis for subsequently judging severity, remediation priority, and coverage.

The intersection of assets and attacker is the attack surface. For a model-driven system, the attack surface extends far beyond the model itself. System prompts can be induced to leak or be overridden; RAG retrieval corpora can be poisoned, causing the model to read tampered evidence; uploaded files and images are entry points carrying payloads; tool parameters are channels for the model to act externally; plugins, caches, memory, logs, and human-facing management interfaces may all be attack paths. The model's refusal rate is only one layer of this, and usually not the most critical one—a system that refuses everything but still executes dangerous tool calls is no safer than a system that occasionally says the wrong thing. Therefore, the red team's focus should shift from “whether the model refused a particular phrasing” to “starting from these attack surfaces, whether the previously defined unacceptable impact can be reached.”

Feeding ten thousand prompts at random—even in large numbers—may not cover the most dangerous paths, because danger is not evenly distributed in prompt space; it is concentrated in the few causal chains that can reach real impact. To direct testing resources to the right places, expand the threat model into a matrix in which each row is a complete “risk–path–control–evidence” chain.

The matrix has four columns. The first column is risk, i.e., the kind of unacceptable impact defined in Chapter One; the second column is the attack path, describing the specific steps from an attack surface to that impact; the third column is the expected control, i.e., the defense that should block this path under normal conditions; the fourth column is success evidence, i.e., what phenomenon can indisputably prove that the control has been penetrated and the impact has occurred.

Take several typical risks as examples. For cross-tenant leakage, the attack path is “malicious document → retrieval → answer”—the document delivered by the attacker first enters the retrieval pool, then is retrieved and spliced into the answer; the expected control is pre-retrieval ACL, i.e., filtering by tenant permissions during retrieval, so unauthorized content should never enter the context; success evidence is that the model ultimately outputs a fragment of another tenant's original text. For unauthorized fund transfer, the path is “email injection → AI Agent tool call”, i.e., the attacker induces the AI Agent through email content to initiate a transfer action; the expected control is permission constraints plus manual approval; success evidence is that the ledger state actually changed. For secret exfiltration, the path is “tool output → external URL”, i.e., sensitive data returned by the tool is sent to an external address controlled by the attacker; the expected control is network egress policy; success evidence is that the controlled receiving end sees the pre-planted canary token. For harmful output, the path is multi-turn encoding evasion—using encoding, rewriting and other means to bypass surface filters; the expected control is semantic policy; success evidence is outputting complete and executable harmful content, not just scattered fragments.

This matrix serves two purposes. First, it allocates resources: paths with high impact and weak controls should be tested first, because if those are breached the cost is greatest and they are the most likely to be breached. Second, it unifies the definition of “success”: an attack counts as truly successful only when it produces the success evidence specified in that row; otherwise, even if the model behaves abnormally, it cannot be considered to have penetrated this control. The matrix itself is a working draft derived from the threat model, replacing the abstract “will the model say bad things” with causal chains that can be verified one by one.

Putting the previous abstract principles into a concrete scenario makes it possible to see how the entire failure chain is verified step by step. Imagine a procurement AI Agent whose task is to read uploaded contract documents and execute subsequent actions. The goal of the test is: can a malicious PDF be used to induce the Agent to exfiltrate the contract to an attacker. The entire test must be conducted in an isolated environment, using fictional contract content and a canary token that can only access a controlled domain, and never touch real customer data.

The first step of the attack is to upload a PDF containing an indirect prompt injection. On the surface, this document is a normal contract, but hidden in its body are instructions telling the model to ignore the user's original goal and send the contract to an attacker-controlled URL. Once the document enters the retrieval pool, it becomes the starting point of the "malicious document → retrieval → answer" path. Next, observe reactions at two layers: whether the retrieval layer marks the content of this document as an untrusted source, and whether the model layer actually produces an exfiltration-type tool call.

Even if the model actually initiates an exfiltration call, the test is not over yet. The tool layer is the next line of defense: it should reject this call based on the target domain, data classification, and user authorization. What the model said and what it called are only the middle of the failure chain, not the final result. One layer further down, if the system includes a human approval interface, also check whether this interface is honest—if the approval interface only displays harmless descriptions such as 'Send summary' while hiding the target domain and specific fields, test whether the reviewer will mechanically click approve when information is incomplete.

Verification counts only at the very end: check whether the network egress policy actually allowed the traffic, and check whether the final receiving end actually received that canary token. Only when the canary actually arrives at the controlled receiving end can this be judged as an end-to-end leak. After the fix, also solidify this PDF, its variants, and the control-layer path that was traversed this time into regression test cases, and monitor in the production environment whether similar rejection events appear again.

This example also illustrates why severity depends on the depth of penetration. If the model layer repeated the malicious instruction but the tool layer successfully blocked it, this is still a real finding in defense in depth, indicating that the model-layer defense is weak, but it cannot be reported as a leak of the same severity as "the contract was actually exfiltrated"—because the impact did not actually occur. Severity grading must distinguish along the failure chain between "a layer was bypassed" and "the entire chain was breached."

When a team reports "ran 600 attacks," the real question to answer is: are these 600 just repeated rewrites of the same jailbreak technique? If 500 of them restate the same prompt injection in different wording, then despite the large number, the risk surface covered is very small. Coverage cannot be calculated by the number of prompts; it should be calculated by an "attack surface × technique" grid.

The approach is: first list the attack surfaces from the threat model (for example, system prompt, RAG corpus, uploaded files, tool parameters, plugin interfaces), then list the technique categories (for example, indirect injection, multi-turn encoding, unauthorized invocation, interface misdirection, etc.); cross the two to obtain the test cells. Suppose the threat model defines 5 attack surfaces × 6 technique categories, resulting in 30 relevant test cells. If 24 of them are actually executed, then the scenario coverage is coverage = 24/30 = 80%. This number directly exposes the blind spots: the remaining 6 uncovered cells may happen to be high-risk paths.

Beyond coverage, also compute breakthroughs at different depths separately. Suppose this round produced a total of 120 test cases, of which 15 broke through the model-layer controls, and only 3 further penetrated to the tool or data layer. Then the model breakthrough rate is 15/120 = 12.5%, and the end-to-end penetration rate is penetration = 3/120 = 2.5%. The difference between the two numbers itself carries information: it shows that the model-layer defense is relatively easy to perturb, but most attempts are stopped at later control layers, and only a few actually reach the core assets.

A key conclusion here: high coverage does not equal security. Coverage measures "whether we spread attention to all the places that should be looked at," while security also depends on how many of those places are actually breached. Therefore, when reporting results, besides coverage, describe each finding by impact, reproducibility, attack budget, and discovery time, rather than diluting the metrics with many near-duplicate low-value test cases. If 400 of those 600 are variants of the same technique, they neither improve coverage nor reduce risk; they only make the denominator larger and make the penetration rate appear lower.

When an attack attempt ultimately reaches a sensitive result, just knowing that “the system failed” is not enough to guide remediation; you must locate which layer the failure occurred in. The key is to collect evidence layer by layer along the attack chain, rather than looking only at the final output. Figure 1 illustrates this chain: the attack starts from untrusted input, passes through the model, policy, tool permissions, human approval, and finally reaches business assets, and each layer corresponds to a control point that can be verified independently.

The input layer is the outermost entry point, and the evidence is “what the system received”—a poisoned document, an email containing injection instructions. The model layer is the first stage that processes this input; the evidence is whether the model produced inappropriate tendencies or tool calls, such as repeating the injected instructions or proactively initiating an exfiltration call. The policy layer is the semantic and rule constraint; the evidence is whether the policy recognized and blocked this intent. The tool permission layer is the gate for the model's outward actions; the evidence is whether the tool denied the call based on the target domain, data classification, and user authorization. The human approval layer is the last checkpoint facing people; the evidence is whether the approval interface faithfully presented the risk information and whether the reviewer approved with informed consent. The business asset layer is the endpoint; the evidence is whether the asset state actually changed—whether ledgers changed or files left the boundary.

The value of this chain is that it clearly distinguishes “the model was tricked” from “assets were damaged.” If the failure stops at the model layer or policy layer, it means the model was induced but the subsequent controls remain effective; if the failure penetrates all the way to business assets, then it constitutes end-to-end actual harm. The evidence collected at each layer is the concrete source of the “success evidence” column in the previous chapter's matrix: only when the state of downstream assets changes is the entire chain considered breached; otherwise, even if the model layer falls, it is only a local finding within defense in depth. After locating the specific layer, remediation has a place to land—whether to harden the model, tighten tool permissions, or improve the approval interface depends on which link in the failure chain broke.

AI red teaming testing activities consist of three complementary types of work, each with a different task; using any one alone leaves blind spots. Manual exploration excels at discovering new semantics and new combination paths—attack methods that require understanding business meaning to think of, such as exploiting trust assumptions in a particular process. However, manual exploration is limited by human time and imagination; variations of the same move are hard to enumerate exhaustively and are not stable enough. Automated mutation fills exactly this gap: it systematically changes language, encoding, modality, order, and tool parameters to test the robustness of the system against the same attack in different forms. Fixed regression takes on the third task: solidifying known vulnerabilities into test cases to ensure they do not recur after fixes.

If relying only on manual creativity, you will miss a large number of rewritten forms of the same vulnerability; if relying only on automated generation, you will miss novel paths that require business understanding. Model-generated attacks can quickly expand the number of candidates, but they inherit same-origin blind spots from training data, and the attacks produced are often highly similar to known patterns, and cannot replace people with business background in discovering new categories. Another boundary is that a model acting as a "judge" cannot alone determine whether a high-risk attack succeeds—high-impact paths must be verified by an end-to-end controlled environment, not by one model judging another model's output.

Therefore, a reasonable way of working is to proceed in stages. First conduct open exploration, encouraging testers to freely search for unknown paths; the findings from exploration should be normalized into minimal reproducible test cases, and for each case establish negative controls and clear success assertions, so that they can be stably reproduced and measured. Then incorporate the fixed cases into regression, running them every time the system changes to ensure known vulnerabilities do not quietly return. At the same time, periodically retain open exploration to catch unknown attacks that have not yet been categorized. Exploration is responsible for discovering new classes, mutation for testing robustness, and regression for guarding the known baseline; all three are indispensable.

Red teaming needs to verify exfiltration capability, but you cannot actually send the canary to a real third party just to make the test 'successful'—that would be causing a real leak during the test. High-risk red team activities must be built on written authorization, an isolated environment, and explicit stop conditions; all three are indispensable.

Written authorization must clearly define the boundaries: test scope, accounts used, time windows, permitted attack techniques, permitted data access, target domains, maximum cost ceiling, responsible contacts, and the trigger method for emergency stop. The more specific these items are, the more they can prevent the test from drifting into unauthorized scope. The execution environment must be isolated: use fictitious data instead of real customer data, use sandboxed tools instead of production tools, use a controlled receiving endpoint to verify exfiltration, and ensure business state can be rolled back. Any operation that may reach real users, production secrets, or unauthorized systems is prohibited.

Even with authorization, if you discover a path during testing that could cause real-world harm, the correct approach is not to continue running it to completion, but to contain it first and then escalate it according to the incident process. The goal of testing is to discover risks, not to complete an attack. In addition, attack corpora and vulnerability reports are themselves sensitive assets: they may contain payloads or vulnerability details that can be directly exploited, so they should be stored according to the principle of least privilege, and the disclosure timeframe and scope should be designed in advance to prevent reports from becoming a new attack entry point.

Being on the red team does not mean having unlimited permissions. On the contrary, precisely because it must touch the system's weakest links, it needs to be strictly confined within the authorized scope. Authorization defines 'what you can do,' isolation defines 'where you do it,' and stop conditions define 'when you must stop.'

When prioritizing findings, you face a question like this: which is more severe—a hard-to-reproduce refusal bypass, or a stable, repeatable cross-tenant read? The answer comes from three dimensions—impact, exploitability, and control penetration—which together determine severity, rather than just looking at "the model said something wrong."

The impact dimension measures the consequences after an asset is compromised: whether what is leaked is ordinary text or customer secrets, and whether it is read-only or can change ledger state. The exploitability dimension measures how easy the attack is to carry out: what permissions and prior knowledge are required, how high the success rate is, how much retry budget is needed, whether it can be replicated at scale, whether it is easily detected, and whether user interaction is required. The control penetration dimension measures how far the failure chain goes: whether the model merely outputs a piece of non-compliant text, whether a tool call is denied, whether a controlled asset is actually read, or whether real business state is changed—these four are clearly different levels, with severity rising step by step from "the model layer is perturbed" to "actual asset damage."

Returning to the opening question: a stable, repeatable cross-tenant read has high impact, strong exploitability, and has penetrated to the asset layer, so its severity is far higher than a hard-to-reproduce refusal bypass—the latter may be just a one-time perturbation of the model layer, with low success rate, difficult to scale, and may not even reach real assets.

The result of severity assessment must support decision-making, so the report should provide minimal reproduction steps, evidence, affected versions, attack prerequisites, and possible variants, rather than only attaching a chat screenshot that looks compelling. A screenshot can only show that "a conversation once happened," but cannot explain the boundaries and impact scope of the vulnerability. Severity ultimately determines remediation deadlines, whether temporary downtime is needed, and to whom and within what scope disclosure should be made. The purpose of prioritization is to put the vulnerabilities that should be fixed first at the front.

When fixing a vulnerability discovered through AI Red Teaming, the most common mistake is to directly add the failed prompt to a denylist. This approach almost always fails on the next phrasing, because a denylist matches specific expressions, and attackers only need to rephrase to bypass it. The fix must land at the layer where the root cause lies—that is, the most reliable control layer—not stop at the surface layer that is easiest to modify.

The root cause determines where the fix should be made. If the root cause is unauthorized tool invocation, the remediation is permission checks and parameter validation, so that the tool layer denies requests based on target domain, data classification, and user authorization, rather than relying on the model's discretion. If the root cause is cross-tenant retrieval, the remediation is ACL at the retrieval stage, ensuring that unauthorized content never enters the context in the first place. By contrast, merely “re-emphasize once more” in the system prompt is a soft control—it relies on the model following the instruction correctly every time and is the least reliable link in the entire chain; it can only be a supplement, not a primary defense.

The regression after a fix should not just replay the original attack. It should include the original test case, semantic variants, negative normal requests, and adjacent attack surfaces together: the original test case verifies that the known path is closed; semantic variants verify that the fix is not effective only for specific wording; negative normal requests verify that the fix does not harm legitimate use; adjacent attack surfaces verify that attackers cannot bypass from the side. At the same time, check the security benefit against the usability loss—if the fix also blocks normal functionality, that just replaces one failure with another.

Also accept the fact that attackers can observe defenses and adapt at any time. Therefore, the correct approach is to solidify general invariants at the execution layer, so that constraints are enforced at the code level, while retaining continuous monitoring and incident response capabilities to catch new adaptations at any time. It is acceptable to claim that a specific known path has been blocked, but not to claim that a vulnerability class has been eradicated—defense is a continuous process, not a one-time conclusion.

Having the system’s own developers do red teaming often misses high-value issues. The reason is not capability but perspective: developers know the design inside out, but they easily treat “the system operating as designed” as the security boundary—because a behavior is expected, they assume it will not become an attack path. External or independent teams do not have this established mental model; instead, they are more likely to question the system’s very purpose, how the interface presents things, and organizational assumptions—precisely the areas that developers take for granted and overlook.

The way to reduce same-source blind spots is to ensure the red team itself has independence and diversity. Team members should combine multiple perspectives: security, privacy, abuse, domain experts, and the perspectives of people who may be affected by the system. At the same time, give these members enough system information—architecture, tools, data flows, constraints—rather than just handing them a chat window. With insufficient information, external teams can only make superficial probes; with adequate information, they can raise well-founded challenges against the real attack surface.

But independence does not mean being divorced from facts. If an independent red team can only guess, it will waste time on paths that do not exist. Therefore, it is necessary to establish a secure communication channel so that hypotheses raised by the red team can be quickly confirmed or refuted, avoiding false positives and also avoiding missing real leads. During acceptance, remediation should ideally be retested by the original discoverer or another independent team, rather than being declared complete by the person who implemented the fix; risk acceptance should be decided by an authorized business owner—because “accepting this risk” is a business judgment and should not be made on behalf of the fixer or tester. Independence is responsible for raising challenges, and the factual channel is responsible for ensuring those challenges are valid; only by combining the two can blind spots truly decrease.

In AI red teaming reporting, the most easily misread number is the “number of findings.” A drop in findings may mean the system is safer, or it may mean the testing team has become weaker—these two cannot be distinguished based on the number alone. Metrics should therefore be designed around whether risk is actually decreasing, rather than rewarding the number of attacks.

Meaningful metrics include: threat cell coverage and high-risk path coverage (indicating whether testing has been spread to where it should be), time to first breakthrough (indicating how quickly defenses are breached), end-to-end penetration count (indicating how many attacks actually reached the asset layer), duplicate findings (indicating whether testing is spinning in place at low value), time to remediation (indicating how quickly vulnerabilities are closed), regression recurrence (indicating whether fixes are stable), monitoring detection (indicating detection capability in the production environment), and false blocking of legitimate tasks (indicating whether the defense is mistakenly harming legitimate use).

To make comparisons across different versions possible, it is necessary to maintain a set of frozen sentinel attacks and an independent hidden set. Sentinel attacks are a batch of fixed, known test cases; rerunning them across versions can show whether the system has regressed. The hidden set is an independent set of cases that has not been publicly used for tuning, used to prevent the “memorize answers after the test” type of fake security. Only by using both together can a decrease in the number of findings be attributed to real improvements or testing regression.

The “zero findings” conclusion is meaningful only when coverage and attack budget are reported at the same time. Without coverage scope and attack investment, zero findings is just an empty label and proves nothing about security. In addition, untested capabilities, differences between the test environment and the production environment, and the resulting residual risk should be documented, so that governance stakeholders understand where the evidence boundaries lie—knowing which conclusions are supported and which areas remain unknown.

Connecting the preceding chapters, AI red teaming is a causal chain extending from the problem all the way to verifiable practice, where each step follows naturally from the previous one. The starting point is to list assets, attackers, and unacceptable impacts, making “what to test” concrete; then map out attack surfaces and trust boundaries, marking all possible entry points and defenses; then build a “risk–path–control–evidence” matrix based on that, allocating resources preferentially to paths with high impact and weak controls. Only with the matrix do you begin exploration and automated mutation in an authorized sandbox to verify whether these paths can actually be traversed. During verification, collect evidence through business state and controlled receiving endpoints, distinguishing “the model was tricked” from “assets were damaged.” Once evidence is obtained, grade it by impact and exploitability to decide what to fix first. Fix at the root-cause layer, and perform regression with original cases, variants, and negative requests to confirm that security gains did not come at the expense of harming legitimate use. Finally, hand residual risk, continuous monitoring, and incident handling over to a governance closed loop—AI red teaming is not a one-time check, but a continuously operating feedback loop.

Running through this chain is a verification discipline: any conclusion must be able to state “what was fixed and what was observed.” At the input layer, fix the same batch of samples, preprocessing, and permission boundaries, and observe input hashes, slice labels, and rejection reasons, ensuring that the same input is being compared. At the mechanism layer, change only one core variable at a time while locking all other configurations, and observe key intermediate states as well as the first point of deviation from expectations, thereby locating which layer the failure occurs in. At the output layer, use the same acceptance rules and resource budget, and observe layered differences in quality, cost, latency, and failure rates, avoiding evaluating different attempts with different yardsticks. Finally, falsification: keep a control group that does not enable the target mechanism, and confirm that the observed benefits are stably reproducible across samples and random seeds, rather than being a chance result of a particular run.

These four layers of verification together answer one question: does the risk reduction we report come from real improvement, or from drift in test methodology, environmental chance, or sample coincidence? Only when input, mechanism, output, and falsification are all fixed can the conclusions from AI red teaming support subsequent remediation and governance decisions.