Human-in-the-Loop and Provenance: Where They Belong (and Where They Break the App)

Ask a room of security people how to make an AI system safe and you will hear "prompt injection" within ten seconds. Ask about human-in-the-loop and provenance and you will get nods — and then, in practice, one of two failures: the team bolts a human-approval step onto everything until the product is too slow to use, or they skip it entirely and let an agent wire money on its own say-so. Both are wrong, for the same reason: HITL and provenance are dials, not switches.

General guidance for security and engineering leaders, not legal advice.

The core idea: match the control to the stakes

A human approval step is expensive. It costs latency, it costs a person's attention, and — the part people forget — it costs credibility. Put a blocking gate in front of a trivial, reversible action and three things happen: the system gets slower, the reviewer starts rubber-stamping because 99% of what they see is fine (alert fatigue), and users learn to route around the gate. You now have a control that looks like protection and delivers none. The same logic governs provenance: a cryptographic signature or an audit trail is worth real effort when being deceived about an artifact's origin is costly, and pure overhead when it is not.

So the question is never "should we have a human in the loop?" It is "which actions, and how heavy a gate?" Here is the register.

Human-in-the-loop: where it belongs

Gate an action when it is irreversible or costly to undo, wide in blast radius, or carries safety / legal / financial weight. Concretely:

• Moving money — payments, payouts, refunds above a threshold, wire transfers.
• Production data changes — deleting or bulk-editing records, schema changes, restores.
• Identity & access — creating or elevating accounts, changing credentials, granting roles.
• Shipping to production — deploys, releases, infrastructure changes, feature flags that affect everyone.
• Mass or public communication — emailing the whole list, posting as the brand, anything a large audience sees.
• Agent tool-calls with real-world side effects — an AI agent that can execute code, call APIs that change state, or spend money.
• Anything you may have to justify — decisions a regulator, auditor or court could question.

For the sharpest end of this list — large payments, destructive production actions — go past single approval to dual-control (two people, like a nuclear key). That is the right amount of friction for an action you genuinely cannot take back.

Human-in-the-loop: where it does not belong

Do not gate high-frequency, low-stakes, easily-reversible actions. Here a human step is a tax, not a safeguard:

• Drafting, summarising, tagging or reformatting internal text.
• A sub-threshold refund or credit within policy.
• Replying to a routine, low-risk support query.
• A reversible config toggle in a non-production environment.
• Read-only lookups and reports.

For these, the right control is automate + log + make reversible: let it run, record who/what/when, and make it one click to undo or sample-review after the fact. That preserves speed and still gives you accountability. Over-gating these is how a well-meaning governance programme produces a product nobody wants to use — and a review queue so noisy the one approval that mattered slips through unread.

Provenance: where it belongs — and where it is overkill

Provenance is knowing, and being able to prove, where something came from: the source of training and retrieval data, whether a piece of content was AI-generated, and an audit trail for what an agent did. It earns its cost when being deceived about origin is expensive:

• Worth it: news and journalism; legal, forensic or compliance evidence; official, financial or safety communications; and the supply chain of your models, prompts and tools, where a tampered artifact is an attack (sign and attest it, the same way you would a software build).
• Overkill: ephemeral, internal, low-trust content — C2PA-signing every throwaway draft, internal thumbnail or scratch note is effort with no payoff. Disclosure ("this was AI-assisted") may still be worth it; a full cryptographic manifest is not.

Note too that provenance has limits worth being honest about: soft watermarks can be stripped, and a signature proves origin, not truth. Match the depth — disclosure, durable watermark, full C2PA manifest with signing — to the stakes.

Find your own line

The register above is the map; these free, in-browser tools are how you apply it to your own system:

• Human-in-the-Loop Decision Advisor — answer a few questions about an action and get a recommendation from "automate + log" to "dual-control", with an explicit warning when you are about to over-gate.
• AI Approval-Gate Calibrator — audit your existing gates: it flags the dangerous actions missing a gate and the trivial ones wasting one.
• Autonomy vs Oversight Lab — practise assigning oversight levels to real agent actions and see both failure modes: the disaster you shipped and the trivial action you strangled.
• AI Provenance Coverage Scorecard — score input, output, decision-trail and supply-chain provenance, with a reminder to match effort to stakes.
• Content Provenance & C2PA Checklist — a practical checklist for Content Credentials, signing, watermarking and disclosure.
• Provenance Chain Lab — build a provenance chain, break a link, and see exactly where verification fails.

Pair this with the voice-agent threat model, the GitHub supply-chain anatomy and the AI security risk mapper — provenance and oversight are the controls that sit under most of those stories.

The takeaway

Human-in-the-loop and provenance belong in the same tier as injection defense — and like any control, they do harm when misapplied. Put human judgement where a mistake is expensive and permanent; automate, log and make reversible everywhere else. Sign and trace what others must be able to trust; don't tax the rest. Calibrated, these controls make an AI system both safer and faster. Uncalibrated, they make it neither.