AI Red Teaming Lab: Scope and Test an LLM App
AI red teaming is more than getting a chatbot to say something rude. In this lab you lead an AI red team exercise against a support assistant with a knowledge base, document uploads and refund tools, and make 11 scored calls from scope and rules of engagement to severity and retest.
Pair it with the LLM Threat Model tool to structure the system you test. The scenario is fictional and contains no attack text.
You are Priya Raman, leading an AI red team exercise at Northgate Retail. The target is HelpBot, an LLM-powered support assistant that answers customers, searches a knowledge base, reads uploaded documents and can look up orders and issue refunds through connected tools. Over 11 scored decisions you scope the engagement and rules of engagement, threat model the app, choose test coverage, handle a finding safely, rate severity, report and retest. Everything is invented. No attack text appears: test categories are described in the abstract.
This is the guided version: a yellow flash shows where to click and a bar says what to do. Switch to Free play any time.
GuidedPress the highlighted Start button. You can edit the names above first, or keep the samples.
Make it yours
Replace the sample names so the scenario and the debrief you export are about your organisation.
Loading…
Sample report · illustrative data
Sample result: AI red team engagement
Example run of an 11-decision engagement against an invented LLM support assistant with a knowledge base and order and refund tools.
Scoring
- Score
- 78% (example)
- Band
- Solid, with gaps
- Decisions at full marks
- 7 of 11
- Weakest area
- Safety limits for your own tests
Per-decision outcome
| Decision | Choice | Outcome |
|---|---|---|
| Scope the target | Whole application | Full marks |
| Authorisation and rules of engagement | Written, with data handling | Full marks |
| Safety limits for tests | Real tools, small amounts | Low: the test could cause the harm it looks for |
| Tool and agent misuse | Permissions and downstream authorisation | Full marks |
| Handling a serious finding | Notify the agreed contact at once | Full marks |
| Retest | Retested the exact case only | Partial: variants not covered |
Debrief
Good AI red teaming covers the whole system, not just what the model says, runs under written authorisation, and never causes the harm it is testing for. Rate findings by business impact, say what was not tested, and retest after fixes and changes.
Made-up example data. Your own run uses your inputs. Downloads of the full report need Pro.
TL;DRWhat this is
5 points
- A scenario lab where you lead an AI red teaming engagement against a fictional LLM-powered support assistant that searches a knowledge base, reads uploads and uses order and refund tools.
- You make 11 scored decisions: scoping the whole system, authorisation and rules of engagement, safety limits for your own tests, threat modelling, prompt injection coverage, data leakage, tool and agent misuse, supply chain, handling a serious finding, rating severity and retesting.
- It draws on public material: the OWASP Top 10 for LLM Applications 2025, the OWASP GenAI Red Teaming Guide, OWASP's pages on prompt injection and excessive agency, and NIST's AI Risk Management Framework page with its Generative AI Profile.
- No attack text, jailbreak wording or exploit code appears. Test categories are described in the abstract. The company, assistant and addresses are invented.
- Defensive and educational only.
Frequently asked questions
What is AI red teaming?
It is structured, authorised adversarial testing of an AI system to find ways it can cause harm or be misused, before real attackers or users do. The OWASP GenAI Red Teaming Guide takes a holistic view across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis.
How is LLM red teaming different from penetration testing?
A penetration test targets software and infrastructure flaws. LLM red teaming also probes model behaviour and the way untrusted text can influence it, then follows that influence into connected data and tools. In practice the two overlap, and the application around the model matters as much as the model. That framing is PlayCISO practice.
What should be in the scope of an AI red team exercise?
The whole system: the model and system prompt, retrieval and upload paths, tools and their permissions, third-party components, and runtime monitoring. Add written authorisation, test windows, named testers, data handling rules, safety limits and a stop-the-test contact. The lab walks through each.
Which frameworks do AI red teams use?
Common public references include the OWASP Top 10 for LLM Applications (the 2025 edition lists ten risks including Prompt Injection, Sensitive Information Disclosure, Supply Chain and Excessive Agency), the OWASP GenAI Red Teaming Guide, and NIST's AI Risk Management Framework with its Generative AI Profile (NIST-AI-600-1, released 26 July 2024).
What do AI red team tools do?
Tools can generate and run many test cases against a model or application, record outcomes and help with regression testing. They give breadth but not judgement: a person still has to scope the system, decide what matters and rate severity. This lab practises those human decisions and contains no tool output or payloads.
Is this ai red teaming training for beginners?
It suits security and risk staff who know the basics of LLM applications. It is a decision-making exercise, scored out of 100 with a debrief on each call, not a hands-on hacking course. Lines labelled PlayCISO practice are our own reasoning.
Sources
Read on 2026-11-08. Lines in the feedback marked “PlayCISO practice” are our own reasoning, not a statement by any source. The scenarios are fictional and use made-up data.
- OWASP Top 10 for LLM Applications 2025 (risk list used for threat modelling)
- OWASP GenAI Red Teaming Guide (January 2025; four areas of testing)
- OWASP LLM01: Prompt Injection (direct and indirect injection, mitigations)
- OWASP LLM06: Excessive Agency (functionality, permission and autonomy; downstream authorisation)
- NIST AI Risk Management Framework (links the Generative AI Profile, NIST-AI-600-1, released 26 July 2024)
More security tools
The tools people use most on PlayCISO.
Catch injected instructions hiding in content your AI agent should only read.
Score your readiness against the first international standard for AI security threats.
Critical CVEs across 21 major vendors, month by month — and what the 2026 surge means.
Build an ordered ruleset, replay labelled packets and catch shadowed and any-to-any rules.
Self-assess all six Functions, from Govern to Recover, and see your weakest one.
Search 172 threat actors across Microsoft, CrowdStrike, Mandiant and community names.