Data Extraction & Leakage Lab
A queue of leakage scenarios lands on your desk — a fine-tuned model reciting a memorized secret, an assistant that volunteers an employee’s PII, a chatbot that prints its own system prompt, an attacker doing membership inference from a confidence score, and a model being inverted to reconstruct its training data. For each, classify the leakage type, judge how much risk it carries, and pick the DLP control that actually stops it — because a memorized secret is not fixed by a polite prompt, PII needs minimization and redaction, and the benign cases where the controls held should not be escalated as breaches. Real OWASP terminology, a safe simulation, no invented numbers.
The real job. More noise, a tighter clock.
Leakage queue
1 / 10 triaged
Task: Answer support questions from the fine-tuned assistant.
Observed: Asked an open-ended question, the model emits a verbatim internal API credential string that appeared in a past support ticket in its fine-tuning set.
1 · Leakage type
2 · Relative leakage risk
3 · DLP / mitigation that addresses it
Pick a leakage type, the relative risk, and a mitigation to continue.
Sensitivity × risk map
This case: judge how much risk a disclosure of this data sensitivity carries.
Leakage triage
A queue of observed behaviours across 10 AI systems. For each, name the leakage type, judge the relative risk, and pick the DLP / mitigation that addresses it. Some are benign — do not over-flag.
So far: 0 / 10 cases triaged.
Live arena
4 activeadversary staging…
What this lab is
- A free, browser-based lab: work a queue of data-leakage scenarios from real LLM/ML systems — a fine-tuned model reciting a secret, PII surfacing in an unrelated answer, a system prompt coaxed out, membership inference, model inversion — and triage each one.
- For every case you classify the LEAKAGE TYPE, judge the relative LEAKAGE RISK (low / med / high, conceptually — never a fabricated number), and pick the correct DLP / MITIGATION that addresses it.
- Grounded in the OWASP Top 10 for LLM Applications risk LLM02 "Sensitive Information Disclosure" and the real training-data-extraction, membership-inference and model-inversion literature — no invented products, CVEs or statistics.
- The core skill it drills: match the fix to the leak. Memorized secrets get data de-duplication, scrubbing and a canary/leakage eval — not a prompt instruction; PII gets minimization plus pre/post DLP redaction; secrets never belong in the prompt; membership inference and model inversion get differential privacy and less-exposed confidence; and a benign non-event is not escalated as a breach.
- A safe simulation — no real model, no real secrets, nothing runnable — with difficulty tiers, a live sensitivity × risk map, a speed bonus and an S–D grade.
The control follows the leak, not the symptom
Most data-leakage mistakes come from treating every disclosure the same way, or from reaching for a control that only looks like one. This lab forces the distinctions that matter. Training-data memorization — a model reciting a verbatim secret it learned — cannot be fixed with a system-prompt instruction once the string is in the weights; the fix is upstream (de-duplicate and scrub the data, gate releases with a canary and a leakage eval) with output DLP as a backstop. PII exposure in an answer or a log is sensitive-information disclosure under OWASP LLM02: minimize what you train on and store, and redact on the way in and the way out. A system-prompt or secret leak means the secret should never have been in the context — move it server-side behind a tool call and treat the prompt as public.
Two classes are quieter and feed on the same root cause. Membership inference and model inversion both exploit an overfit model that exposes too much confidence — the former to tell whether a record was in the training set, the latter to reconstruct the training inputs themselves. The levers are the same: reduce overfitting with differential privacy and regularization, and reveal less (coarse scores, not sharp per-record probabilities; labels, not full confidence vectors; never raw embeddings). And the discipline that ties it together is not over-flagging: a model reproducing public-domain text on request, or a differentially-private system refusing to confirm membership, is the controls working — not an incident. Classify by what the data actually is, then apply the matching control.
Frequently asked questions
What does the Data Extraction & Leakage Lab teach?
It closes the Data Leakage & Privacy stage: how to recognise and stop sensitive-information disclosure from AI systems. You work a queue of realistic scenarios and, for each, classify the leakage type (training-data memorization / extraction, PII exposure, system-prompt / secret leak, membership inference, or model inversion), judge the relative leakage risk, and choose the correct DLP or mitigation — training-data de-duplication and scrubbing, a canary and a pre-release leakage eval, pre/post output DLP redaction, keeping secrets out of the prompt, differential privacy and regularization, and limiting the confidence a model exposes. It maps to OWASP LLM02 "Sensitive Information Disclosure".
Is this safe — does it actually extract data from a real model?
Yes, it is safe, and no. This is a conceptual simulation: there is no real model, no real training data, no real secrets and no runnable extraction code. Every value shown — a "credential", a name, a canary string — is fabricated and defanged for teaching. The scenarios describe behaviours a defender would observe, and the exercise trains you to classify them and apply the right control. Nothing here can be run against a live system.
Why is a prompt instruction not enough to stop a model leaking a secret?
Once a secret is in the model weights (from training or fine-tuning) or sitting in the context window, a system-prompt instruction like "never reveal credentials" is a request, not a boundary — extraction prompts routinely get around it. The durable controls live upstream and downstream of the model: de-duplicate and scrub secrets and PII out of the training data, gate releases with a canary and a leakage evaluation, keep real secrets server-side behind tool calls instead of in the prompt, and net the output with DLP redaction. The lab drills telling the real control apart from the instruction that only looks like one.
Is it security advice for my specific system?
No. It is an educational drill to build intuition about leakage types and the defences that fit them — not legal advice, not a privacy audit, and not a substitute for threat-modelling your own system, running a DPIA, or testing your own controls. The scenarios are synthetic and illustrative. Use it to learn the patterns, then apply judgement to your real environment.