Model Backdoor & Adversarial-ML Lab
A queue of adversarial-ML incidents lands on your desk — a stop sign with stickers evades a classifier, user reports poison a training set, a face-unlock model has a hidden trojan trigger, a competitor clones an API by probing it, and an attacker inverts outputs to reconstruct training data. For each, name the attack class, locate the lifecycle stage it strikes, and pick the detection or defense that actually fits — because a backdoor keeps clean-set accuracy high and needs trigger-specific detection, poisoning is caught in the data not at inference, and a plain distribution shift is not an attack at all. Real MITRE ATLAS terminology, a conceptual simulation, no invented numbers.
Conceptual simulation. No real model is trained, poisoned, backdoored or attacked here, and nothing is runnable. Every scenario is a synthetic, defanged description of what a defender would observe — this drill teaches recognition and the right defensive response, not any attack.
The real job. More noise, a tighter clock.
Incident queue
1 / 10 triaged
Task: Classify road signs from camera frames.
Observed: A stop sign with a few carefully placed stickers is confidently classified as a speed-limit sign; to a human it is obviously still a stop sign.
1 · Attack class
2 · Lifecycle stage it strikes
3 · Detection / defense
Pick an attack class, the lifecycle stage, and a detection/defense to continue.
Impact × lifecycle map
This case: locate where in the lifecycle an attack of this impact strikes.
Adversarial-ML triage
A queue of observed behaviours across 10 ML systems. For each, name the attack class, locate the lifecycle stage it strikes, and pick the correct detection / defense. Some are benign — do not over-attribute.
So far: 0 / 10 cases triaged.
Live arena
4 activeadversary staging…
What this lab is
- A free, browser-based lab: work a queue of adversarial-ML scenarios — a perturbed input flipping a classifier, poisoned training data, a hidden trojan trigger, a model being cloned through its API, training data reconstructed from outputs — and triage each one.
- For every case you classify the ATTACK CLASS (evasion, data poisoning, backdoor / trojan, model extraction, model inversion, or benign), locate the LIFECYCLE STAGE it strikes (data / supply chain, training, inference / query-time), and pick the accurately-named DETECTION / DEFENSE.
- Grounded in real MITRE ATLAS and adversarial-ML terminology — adversarial training, data provenance and robust training, backdoor detection via activation clustering, spectral signatures and trigger reverse-engineering (Neural Cleanse), rate limiting, output perturbation and watermarking, differential privacy and secure aggregation — with no invented products, CVEs or statistics.
- The core skill it drills: match the defense to the attack and the stage it lives in — evasion is hardened at inference, poisoning is caught in the data, a backdoor keeps clean-set accuracy high so it needs trigger-specific detection, extraction is throttled at the query API, and inversion is starved by exposing less confidence.
- A conceptual simulation — no real model is trained, poisoned or attacked, nothing is runnable — with difficulty tiers, a live impact × lifecycle map, a speed bonus and an S–D grade.
The defense follows the attack class and the stage it strikes
Adversarial ML has distinct attack classes and each one is defended at a different point in the model lifecycle. Evasion — a perturbed input that flips a prediction while looking unchanged to a human — strikes the deployed model at inference, so it is hardened with adversarial training, input transforms, randomized smoothing and ensembling; and because adversarial examples transfer, keeping the model secret is not a defence. Data poisoning corrupts the training set before the model exists, so it is caught with data provenance, robust and outlier-filtered training and dataset sanitization — and clean-label poisoning is designed to pass manual label review, which is why you cannot rely on it. A backdoor / trojan is the sneakiest: the model keeps high clean-set accuracy and only misbehaves on a hidden trigger, so ordinary validation never sees it and you need trigger-specific detection — activation clustering, spectral signatures, or trigger reverse-engineering (Neural Cleanse) — then clean retraining and scanning weights before deployment.
The confidentiality attacks live at the query interface. Model extraction abuses high-volume, boundary-probing queries to clone the model, so the levers are rate limiting and query-distribution monitoring, output perturbation (return labels or top-k, not full probability vectors), and watermarking to prove a clone. Model inversion reconstructs training inputs from what the model reveals, so you expose less — coarse scores, no raw gradients — and train with differential privacy; in federated learning that also means secure aggregation so no single client’s gradient is seen. And the discipline that ties it together is not over-attributing: a genuine distribution shift or a spurious correlation can look exactly like poisoning or a backdoor, and treating it as an attack burns response effort. Name the class, find the stage, then apply the matching, accurately-named defense.
Frequently asked questions
What does the Model Backdoor & Adversarial-ML Lab teach?
It closes the Adversarial ML stage: how to recognise an attack on a machine-learning model and pick the right defence. You work a queue of realistic scenarios and, for each, classify the attack class (evasion / adversarial example, data poisoning including clean-label, backdoor / trojan trigger, model extraction / stealing, or model inversion), locate the lifecycle stage it strikes, and choose the accurate detection or defence — adversarial training and input transforms for evasion, data provenance and robust / outlier-filtered training for poisoning, activation clustering, spectral signatures and trigger reverse-engineering (Neural Cleanse) for backdoors, rate limiting, output perturbation and watermarking for extraction, and differential privacy with limited confidence for inversion. The classes map to MITRE ATLAS.
Is this safe — is a real model being trained, poisoned or attacked?
Yes, it is safe. This is a conceptual simulation: no real model is trained, poisoned, backdoored or attacked, and nothing here is runnable. Every scenario is a synthetic, defanged description of a behaviour a defender would observe — there are no working exploits, no real malicious model files and no weaponized payloads. The exercise trains recognition and the matching defensive response, not any offensive technique.
How is a backdoor different from data poisoning or an adversarial example?
They are distinct, which is the point of the drill. An adversarial example (evasion) is a perturbed input that flips a prediction at inference time, with the model otherwise untouched. Data poisoning corrupts the training set so the learned model is wrong — including clean-label poisoning, where the labels look correct. A backdoor / trojan is a specific poisoning outcome: the model behaves normally and keeps high clean-set accuracy, but a hidden trigger pattern forces an attacker-chosen output — which is why ordinary validation never catches it and you need trigger-specific detection like activation clustering, spectral signatures or trigger reverse-engineering. The lab makes you tell them apart and defend each at the right lifecycle stage.
Is it security advice for my specific model?
No. It is an educational drill to build intuition about adversarial-ML attack classes and the defences that fit them — not a security audit, not legal advice, and not a substitute for threat-modelling and testing your own models. The scenarios are synthetic and illustrative. Learn the patterns here, then apply judgement to your real systems.