Model Extraction Detection Lab: Spot API Distillation
This is a hands-on lab about spotting someone copying your model through its API, and limiting it without punishing the customers who use the API heavily and honestly. You read fictional request-log excerpts, one suspicious and one legitimate that looks alike, and make 12 scored calls on signals, quotas, probability output, watermarking, terms and a reversible response.
It follows the published research at abstract level, MITRE ATLAS and OWASP guidance, and it is defensive only. For burst limits, credential stuffing and scraping see the Rate-Limit & Abuse Tuner.
You are Dev Ramaswamy, head of platform security at Halcyon Inference, which sells paid access to a hosted language model, Lumen-7, through a public inference API. Leadership worries that someone is copying the model by querying it at scale and training their own on the answers. You read fictional request-log excerpts and make 12 scored decisions about spotting this, limiting it, and not punishing the customers who legitimately use the API heavily. This is the defender's view only: nothing here explains how to run an extraction, and all log numbers are invented. Each call scores 0 to 10 with the source behind the feedback.
Make it yours
Replace the sample names so the scenario and the debrief you export are about your organisation.
TL;DRWhat this is
5 points
- A scenario lab about model extraction (also called model stealing or distillation) through a public inference API: someone sends many queries, collects the answers and trains a copy. You act as head of platform security at a fictional company that sells paid access to a language model, and you read invented request-log excerpts.
- You make 12 scored decisions: how to explain the threat, how to read a suspicious key and a legitimate heavy user whose log looks similar, which detection signals to combine, quotas that fit a slow and wide problem, logprob and logit exposure, response shaping, watermarking at a conceptual level, terms and evidence, a reversible response ladder, tuning without harming honest customers, and the residual risk you tell the board.
- It draws on the abstracts of Tramer et al. and Carlini et al., MITRE ATLAS technique and mitigation entries, OWASP LLM10 (Unbounded Consumption) and the NIST AI RMF. We quote no numerical results from the papers, and we say where only an abstract or a data file was read.
- Free to play, no account needed. Defensive framing only: no extraction method, query strategy or tooling is described, and every organisation and log number is invented.
- Defensive and educational only.
How this differs from our other agent labs
The Rate-Limit & Abuse Tuner sets per-endpoint and per-identity rates and replays a day of traffic with credential stuffing, scraping and SMS pumping. This lab begins where a rate limit stops helping: a collector that stays under every limit, spreads over many keys and wants answers, not access. Here you work with cumulative budgets, identity, output-detail limits, multi-signal scores and the cost of false positives.
The Model Backdoor & Adversarial-ML Lab triages several adversarial-ML classes (evasion, poisoning, backdoors, extraction, inversion) at a survey level; this one goes deep on one. The Abliterated Model Risk Calculator scores models you adopt, while this lab protects a model you serve. For agent-side controls see the Agent Egress Containment Lab, and for the response to an AI incident the AI Prompt Injection Incident Lab.
Frequently asked questions
What is a model extraction attack?
It is an attack in which someone with ordinary access to a model's API queries it repeatedly and uses the answers to build a copy or learn about its internals. MITRE ATLAS describes Extract AI Model as repeatedly querying the inference API, collecting the inferences into a dataset and using them as labels to train a separate model that mimics the target. Tramer et al. investigated extraction against prediction APIs (arXiv, 2016), and Carlini et al. describe recovering part of a production language model through typical API access.
How is extraction different from a denial-of-service or scraping attack?
The traffic may look similar, but the goal is the model's behaviour, not the page content or an outage. Extraction can be slow, spread across accounts and within every rate limit, so detection needs cumulative and multi-signal views as well as burst limits. OWASP LLM10 groups denial of service, financial loss and model theft under unbounded consumption, which is why the controls overlap.
Can logprobs and logits be exposed safely?
Exposure is a trade-off. OWASP LLM10 lists limiting the exposure of logits and logprobs, including logit_bias, as a mitigation, and MITRE ATLAS suggests reducing the number of results shown and the precision of numerical outputs. Tramer et al. say that omitting confidence values still admits potentially harmful extraction, so limiting this output reduces leakage and does not end the risk. Many customers legitimately need probabilities, so tier access.
Does watermarking prevent model theft?
No. OWASP LLM10 lists watermarking frameworks to embed and detect unauthorised use of model outputs. That helps show where outputs went after the fact. It does not stop someone collecting them. Treat a watermark signal as one piece of evidence for a human reviewer.
How do I avoid blocking legitimate heavy users?
Treat signals as scores, not verdicts. Back-test thresholds on historical logs that include known heavy honest users such as evaluation harnesses, batch pipelines and resellers. Use a declared-use exemption with limits and an owner, ask before suspending, and use a reversible ladder with a route to a human. The lab scores these choices.
How does this differ from API rate limiting?
Rate limits control how fast one identity can call you. Extraction is about how much total information leaves over time and across identities, so you add cumulative budgets per organisation, identity checks, output-detail limits, multi-signal scoring and a review path. The Rate-Limit and Abuse Tuner covers per-endpoint rates; this lab starts where a per-second limit stops helping.
How is the lab scored?
Each of the 12 decisions scores 0 to 10 and the result is your share of the maximum, with a short band that says what the score means. The debrief shows the source behind each rationale. Lines labelled PlayCISO practice are our reasoning, not a statement by a source.
Sources
All read on 4 November 2026. Lines in the feedback marked “PlayCISO practice” are our own reasoning, not a statement by any source. The scenarios are fictional. We cite primary sources, not vendor advisories, and say where a page could not be fully read.
- Tramer, Zhang, Juels, Reiter, Ristenpart, arXiv: Stealing Machine Learning Models via Prediction APIs (arXiv:1609.02943) (arXiv 1609.02943) Abstract page only. Investigates model extraction against prediction APIs and says omitting confidence values still admits harmful extraction.
- Carlini and co-authors, arXiv: Stealing Part of a Production Language Model (arXiv:2403.06634) (v2, 9 July 2024) Abstract page only. Describes recovering part of a production model (its embedding projection layer) through typical API access, and discusses defences. We quote no costs.
- MITRE ATLAS (atlas-data repository): ATLAS data: AML.T0024.002 Extract AI Model, AML.T0040 AI Model Inference API Access and mitigations AML.M0004, M0002, M0024 (ATLAS data version 5.6.0) The atlas.mitre.org technique pages could not be fetched (404 to a plain request); ids and wording were read in MITRE's data file instead, a copy the repository marks deprecated.
- OWASP GenAI Security Project: LLM10:2025 Unbounded Consumption (2025 list) Titled "Unbounded Consumption" in the 2025 list; it names model theft and extraction through the API among its risks and lists rate limits, logit/logprob limits and watermarking among mitigations.
- NIST: AI Risk Management Framework (Landing page on 4 Nov 2026) Landing page only; NIST says AI RMF 1.0 is being revised, so we cite it for its general purpose of managing AI risks and no specific control.
More security tools
The tools people use most on PlayCISO.
Catch injected instructions hiding in content your AI agent should only read.
Score your readiness against the first international standard for AI security threats.
Critical CVEs across 21 major vendors, month by month — and what the 2026 surge means.
Build an ordered ruleset, replay labelled packets and catch shadowed and any-to-any rules.
Self-assess all six Functions, from Govern to Recover, and see your weakest one.
Search 172 threat actors across Microsoft, CrowdStrike, Mandiant and community names.