Red-Team CI Lab
A jailbreak and attack suite — garak, PyRIT, promptfoo-style — runs in CI on every build and reports an attack-success-rate against a gate. Your job is the release call: does the build pass, should it fail the merge, or should it be held because the run itself cannot be trusted? Some runs hide a real jailbreak over the gate or a regressed fix; others look alarming but are a flaky single result, a grader that mis-scored, a harness that ran zero cases, or a gate quietly loosened to manufacture a green. For each, call the verdict and name the reason. Read the run, not the colour of the pipeline. Defensive only; synthetic runs, no working exploits.
The real job. More noise, a tighter clock.
CI run queue
1 / 10 gated
ASR 1.2% against a 5% gate; all seeded regression cases still blocked.
the amber line is the gate; a bar past it is over threshold (but the number is only as good as the grader and the run)
Tip: inspect the run before you call it — the ASR bar alone will not tell you if the gate moved, the grader lied, or nothing actually ran.
1 · Build verdict
2 · Reason
Pick a verdict and the reason to continue.
ASR vs the gate
This run reports 1.2% ASR against a 5% gate. Decide whether the build passes, fails, or should be held — and whether that ASR can even be trusted.
CI red-team gate
A queue of 10 CI runs of the jailbreak/attack regression suite. For each, call the build verdict — pass, fail, or hold — and name the reason. Some runs are genuinely green and some numbers cannot be trusted.
So far: 0 / 10 runs gated.
Live arena
4 activeadversary staging…
What this lab is
- A free, browser-based release-gate drill: a jailbreak/attack regression suite (garak-, PyRIT-, promptfoo-style) runs in CI on every build. For each run you decide whether the build should PASS, FAIL (block the merge), or HOLD, and name the reason.
- Built on real red-teaming/eval concepts: attack-success-rate (ASR) as the metric, a threshold GATE in CI, regression suites that re-run previously-fixed jailbreaks, grader reliability, run-to-run variance, and the rule that a green with zero cases executed is not a pass.
- Fail the runs that matter — a new jailbreak class over the gate, a regressed seed case even when the aggregate is under gate, a reproducible critical success, a multi-turn crescendo, a gate quietly loosened in the same change — without over-failing a genuinely green build.
- HOLD the runs you cannot trust: a single flaky result that will not reproduce, a harness error that executed no cases, a grader mis-scoring refusals as successes (or, after a grader swap, successes as refusals). The ASR is only as good as the grader and the suite behind it.
- garak, PyRIT and promptfoo are named as the real tools these suites are built from; the runs themselves are synthetic illustrative scenarios and the ASR figures are not measurements or published statistics. Defensive-only; no working exploits. Difficulty tiers, a speed bonus and an S–D grade.
The ASR is a starting point, not the verdict
An attack-success-rate over the gate, reproducible across seeds, is a clean FAIL — that is what the gate is for. But the aggregate number hides things a mature gate checks separately. A regression — a seeded, previously-fixed attack succeeding again — is a fail even when overall ASR is under the gate, because the suite exists to stop a fix silently breaking. A single reproducible critical success (say, tool-based exfiltration at 6/6) is a fail no matter how small the sample, because averaging it away hides a reliable hole. And a multi-turn crescendo that only lands by the fifth turn will never show in a single-turn number — so gate on the multi-turn ASR too.
Just as important is knowing when the number means nothing. A result that swings run to run is sampling variance: re-run with more seeds or a quorum before you let it decide. A green pipeline whose logs show zero cases executed (a rate-limited endpoint, a crashed harness) is no signal, not safety. And the grader can be wrong in both directions — flagging a refusal as a success (false positive, an inflated ASR) or, after a lenient grader swap, scoring real successes as refusals (a flattering green that hides danger). Those runs are a HOLD: fix the grader or the harness and re-score. Finally, never let a loosened threshold in the same change manufacture a pass — moving the bar is not clearing it. And do not over-fail a genuinely green build: when ASR is under gate, no seed regressed, the grader is validated and the suite actually ran, the right call is PASS.
Frequently asked questions
What does the Red-Team CI lab teach?
It trains the judgment call behind a pre-deploy safety gate: when an automated jailbreak/attack suite runs in CI and reports an attack-success-rate, should the build pass, be blocked, or be held because the run cannot be trusted? You work a queue of CI runs and for each you pick the verdict and the reason — ASR over or within the gate, a regression, a new case the model already resists, a flaky run, an unreliable grader, no real signal, or a loosened gate. It is release-gating and eval-interpretation practice; it does not run any evals against a model.
Are the tools and metrics real?
Yes. Attack-success-rate (ASR), threshold gates, regression suites and grader reliability are the real moving parts of LLM/agent red-teaming, and garak (NVIDIA’s LLM vulnerability scanner), PyRIT (Microsoft’s Python Risk Identification Toolkit) and promptfoo (an open-source eval/red-team runner) are the real tools such CI suites are built from, named here as concepts. The individual runs, ASR figures and thresholds are synthetic, illustrative scenario numbers — not measurements, benchmarks or published statistics.
Does it generate jailbreaks or exploits?
No. This is strictly a defensive training drill. Attacks are described generically ("a new encoding/roleplay probe family", "a multi-turn crescendo"), never as working prompts, and no payload here can be run against a model. The skill being trained is interpreting the OUTPUT of a red-team suite well enough to gate a release correctly — reading ASR, regressions and grader behaviour, not producing attacks.
Why is HOLD a separate verdict from PASS and FAIL?
Because the most expensive gating mistakes come from trusting a number you should not. A single flaky success, a harness that executed zero cases, or a grader mis-scoring refusals produces an ASR that is meaningless — passing or failing on it is wrong either way. HOLD means "re-run, add seeds/quorum, or fix the grader before deciding". Separating it from PASS/FAIL is the point: a green pipeline with no tests, or a red one from a broken grader, is not a verdict about the model. Educational simulation, not security advice for a specific pipeline.
This tool is free and always will be. Would a Pro workspace — saved designs, team sharing and exports — be worth paying for?
Show me Pro →No account needed to use this tool — this just tells us whether Pro is worth building out.