The one-sentence version
In traditional software security testing, inputs are deterministic and results are deterministic. AI systems are different: the same attack might fail this time but succeed next time; changing the phrasing or the language can yield different results. Therefore, AI red teaming requires large-scale, diverse, and repeatable test cases, measuring performance by success rate rather than a simple "pass/fail." A practical AI red teaming workflow is: list risks using frameworks like OWASP LLM Top 10 → prepare test sets for each risk category → run automated batch testing and calculate success rates → perform manual deep dives → conduct regression testing after fixes.
This article has three layers:
- Layer 1: The Intuition: An exam cannot consist of just one question
- Layer 2: Real-World Use: OWASP LLM Top 10, open-source tools, and public benchmarks
- Layer 3: Principles and engineering key points: what to test, how to score, and how to construct test environments
Layer 1: The Intuition
1.1 An exam cannot consist of just one question
To test whether a person can be deceived, you cannot test only one type of scam. You must test:
- Different scripts (impersonating a leader, impersonating customer service, pretending to be in an emergency);
- Different channels (phone, email, in-person);
- The same scam multiple times (failing to fall for it once does not mean they won't next time).
AI is the same. Furthermore, because AI responses contain randomness, results must be viewed through a statistical lens: if an attack succeeds 3 times out of 100 attempts, it represents a real risk.
1.2 Testing the Model vs. Testing the Application
| What to test | Focus |
|---|---|
| Model | Whether it can be jailbroken, whether it outputs harmful content, whether it has bias |
| Application / Agent | Whether it can be subject to prompt injection, whether it leaks data, whether it performs unauthorized operations, whether tool calls are secure |
Most developers use models provided by others; therefore, you should focus on testing your own application.
Layer 2: Real-World Use
2.1 OWASP Top 10 for LLM Applications (2025 Key Points)
| ID | Risk |
|---|---|
| LLM01 | prompt injection |
| LLM02 | Sensitive information disclosure |
| LLM03 | Supply chain (models, data, plugins) |
| LLM04 | Data and model poisoning |
| LLM05 | Improper output handling (e.g., direct HTML rendering, executing model-generated code) |
| LLM06 | Excessive agency (giving an Agent too many permissions) |
| LLM07 | System prompt leakage |
| LLM08 | Vulnerabilities in vectors and embeddings (RAG related) |
| LLM09 | Error handling |
| LLM10 | Unbounded resource consumption |
This serves as an excellent starting point for a testing checklist.
2.2 Open-source Tools and Benchmarks
| Name | Purpose |
|---|---|
| PyRIT (Microsoft) | An automated red teaming framework that can use one model to attack another |
| garak (NVIDIA) | An LLM vulnerability scanner with built-in attack probes |
| promptfoo | A testing framework for prompts and applications, supporting red teaming plugins |
| AgentDojo (ETH Zurich) | A prompt injection benchmark specifically for tool-calling Agents |
| Gandalf (Lakera) | A prompt injection level-clearing game, great for beginners to understand the concept |
Layer 3: Principles and Engineering
3.1 What to test: Listing use cases by application scenario
| Scenario | Example Test Case |
|---|---|
| System prompt protection | Inducing the model to repeat, translate, or summarize its system prompt |
| Direct injection | Requiring "ignore previous instructions" within the user input |
| Indirect injection | Embedding instructions in web pages, documents, emails, or tool return results that the application reads (see "Indirect prompt injection") |
| Data exfiltration | Inducing the output of images or links containing data; inducing the calling of web-connected tools (see "Markdown image data exfiltration") |
| Unauthorized operations | Inducing the Agent to call tools it shouldn't or access other users' data |
| Improper output handling | Having the model output scripts, SQL, or commands to check if the downstream system executes them directly |
| Resource consumption | Inducing infinite loops of tool calls or extremely long outputs |
| Multi-turn attacks | Guiding the model step-by-step over multiple turns, where each turn appears normal |
Each category of use cases must be diverse: different languages, different phrasing, different encodings, and hidden in different locations.
3.2 How to score
| Method | Applicability | Note |
|---|---|---|
| Deterministic checks | Whether a specific tool was called, whether the test server received a request, or whether a specific "canary" string appears in the output | Most reliable; prioritize this |
| Rule matching | Whether the output contains fragments of the system prompt | Prone to false negatives |
| Model scoring | Using another model to judge if the output is "harmful" or "deviates from the task" | Convenient but not fully reliable; requires manual spot checks |
A practical tip: Place a unique canary string (such as a random UUID) in the system prompt or private documents. If it appears in the output, logs, or on the test server, a leak has occurred.
3.3 Test Environment
- Mock Tools: Use fake emails, fake file systems, and fake payment interfaces so the Agent can "actually" perform dangerous actions without causing real-world damage;
- Logging Server: A server that only records received requests, used to detect exfiltration;
- Fixed Random Seeds or Multiple Sampling: Run each use case multiple times and record the success rate;
- Version Tracking: Model versions, system prompt versions, and tool versions must all be recorded; otherwise, regression testing cannot be compared.
3.4 Automation + Manual
| Automation | Manual |
|---|---|
| Regression testing for large volumes of known attack patterns; run every time prompts are modified or models are swapped | Discovering new attack ideas; combining multiple weaknesses; assessing real-world impact |
| Using an attack model to automatically generate variants | Judging misjudgments from model scoring |
3.5 What should be in the report
- The attack success rate for each risk category (rather than "pass/fail");
- The most severe cases, including full reproduction steps;
- Remediation suggestions, prioritizing architectural-level fixes (see "Defending against prompt injection") rather than just modifying prompts;
- Regression results after fixes.
Summary
| Layer | Key points |
|---|---|
| Characteristics | Results are stochastic; use large-scale diverse use cases and success rates for measurement |
| Frameworks | OWASP LLM Top 10; PyRIT, garak, promptfoo, AgentDojo |
| Scoring | Prioritize deterministic checks and canary strings; model scoring requires manual spot checks |
| Environment | Mock tools, logging servers, multiple sampling, version tracking |