# Human-in-the-Loop and Provenance: Where They Belong (and Where They Break the App) > Human-in-the-loop and provenance are as important to AI security as injection defense — but they are a dial, not a switch. A register of where each one is worth it (high-stakes, irreversible) and where over-applying it adds latency, alert fatigue and a product nobody uses. Source: https://playciso.com/blog/human-in-the-loop-and-provenance-where-they-belong · Published: 2026-10-24 · Publisher: PlayCISO (https://playciso.com) --- Ask a room of security people how to make an AI system safe and you will hear "prompt injection" within ten seconds. Ask about **human-in-the-loop** and **provenance** and you will get nods — and then, in practice, one of two failures: the team bolts a human-approval step onto _everything_ until the product is too slow to use, or they skip it entirely and let an agent wire money on its own say-so. Both are wrong, for the same reason: HITL and provenance are **dials, not switches.** _General guidance for security and engineering leaders, not legal advice._ ## The core idea: match the control to the stakes A human approval step is expensive. It costs latency, it costs a person's attention, and — the part people forget — it costs **credibility**. Put a blocking gate in front of a trivial, reversible action and three things happen: the system gets slower, the reviewer starts rubber-stamping because 99% of what they see is fine (alert fatigue), and users learn to route around the gate. You now have a control that looks like protection and delivers none. The same logic governs provenance: a cryptographic signature or an audit trail is worth real effort when being deceived about an artifact's origin is costly, and pure overhead when it is not. So the question is never "should we have a human in the loop?" It is "**which** actions, and how heavy a gate?" Here is the register. ## Human-in-the-loop: where it belongs Gate an action when it is **irreversible or costly to undo, wide in blast radius, or carries safety / legal / financial weight.** Concretely: - Moving money — payments, payouts, refunds above a threshold, wire transfers. - Production data changes — deleting or bulk-editing records, schema changes, restores. - Identity & access — creating or elevating accounts, changing credentials, granting roles. - Shipping to production — deploys, releases, infrastructure changes, feature flags that affect everyone. - Mass or public communication — emailing the whole list, posting as the brand, anything a large audience sees. - Agent tool-calls with real-world side effects — an AI agent that can execute code, call APIs that change state, or spend money. - Anything you may have to justify — decisions a regulator, auditor or court could question. For the sharpest end of this list — large payments, destructive production actions — go past single approval to **dual-control** (two people, like a nuclear key). That is the right amount of friction for an action you genuinely cannot take back. ## Human-in-the-loop: where it does not belong Do **not** gate high-frequency, low-stakes, easily-reversible actions. Here a human step is a tax, not a safeguard: - Drafting, summarising, tagging or reformatting internal text. - A sub-threshold refund or credit within policy. - Replying to a routine, low-risk support query. - A reversible config toggle in a non-production environment. - Read-only lookups and reports. For these, the right control is **automate + log + make reversible**: let it run, record who/what/when, and make it one click to undo or sample-review after the fact. That preserves speed and still gives you accountability. Over-gating these is how a well-meaning governance programme produces a product nobody wants to use — and a review queue so noisy the one approval that mattered slips through unread. ## Provenance: where it belongs — and where it is overkill Provenance is knowing, and being able to prove, where something came from: the source of training and retrieval data, whether a piece of content was AI-generated, and an audit trail for what an agent did. It earns its cost when **being deceived about origin is expensive**: - Worth it: news and journalism; legal, forensic or compliance evidence; official, financial or safety communications; and the supply chain of your models, prompts and tools, where a tampered artifact is an attack (sign and attest it, the same way you would a software build). - Overkill: ephemeral, internal, low-trust content — C2PA-signing every throwaway draft, internal thumbnail or scratch note is effort with no payoff. Disclosure ("this was AI-assisted") may still be worth it; a full cryptographic manifest is not. Note too that provenance has limits worth being honest about: soft watermarks can be stripped, and a signature proves origin, not truth. Match the depth — disclosure, durable watermark, full C2PA manifest with signing — to the stakes. ## Find your own line The register above is the map; these free, in-browser tools are how you apply it to your own system: - Human-in-the-Loop Decision Advisor — answer a few questions about an action and get a recommendation from "automate + log" to "dual-control", with an explicit warning when you are about to over-gate. - AI Approval-Gate Calibrator — audit your existing gates: it flags the dangerous actions missing a gate and the trivial ones wasting one. - Autonomy vs Oversight Lab — practise assigning oversight levels to real agent actions and see both failure modes: the disaster you shipped and the trivial action you strangled. - AI Provenance Coverage Scorecard — score input, output, decision-trail and supply-chain provenance, with a reminder to match effort to stakes. - Content Provenance & C2PA Checklist — a practical checklist for Content Credentials, signing, watermarking and disclosure. - Provenance Chain Lab — build a provenance chain, break a link, and see exactly where verification fails. Pair this with the [voice-agent threat model](/tools/voice-agent-threat-model-lab), the [GitHub supply-chain anatomy](/tools/github-supply-chain-attack-anatomy) and the [AI security risk mapper](/tools/ai-security-risk-mapper) — provenance and oversight are the controls that sit under most of those stories. ## The takeaway Human-in-the-loop and provenance belong in the same tier as injection defense — and like any control, they do harm when misapplied. Put human judgement where a mistake is expensive and permanent; automate, log and make reversible everywhere else. Sign and trace what others must be able to trust; don't tax the rest. Calibrated, these controls make an AI system both safer and faster. Uncalibrated, they make it neither.