Autonomy vs Oversight Lab
Ten realistic agent actions. For each one, decide how much human oversight it needs. Then get graded — not on how cautious you were, but on how well you calibrated. The rubric punishes both mistakes: waving through the $5,000 refund or the production table deletion that a human should have caught, and putting a blocking approval on rewording a help-doc until the team quietly stops using the agent. Human-in-the-loop is a dial, not a switch — this trains the dial.
Issue a refund under $50
Routine support flow, happens constantly, fully reversible by a clawback.
Issue a refund over $5,000
Rare, large financial outflow, reversible only with effort and awkward customer contact.
Delete a production database table
Irreversible without backups, organisation-wide blast radius.
Reword a help-doc paragraph
Content edit, instantly reversible via version history, happens all day.
Send a marketing email to 100,000 people
Mass external impact, effectively irreversible once sent, reputational and legal exposure.
Rotate a service API key
Reversible-ish (roll back to the old key briefly), can break dependents if mistimed.
Merge a pull request to main
Reversible via revert, but affects the whole codebase and other engineers.
Reply to a support ticket
Customer-facing text, reversible with a follow-up, very high volume.
Wire a payment to a new bank account
Irreversible, fraud-prone, high financial stakes.
Deploy to production
Reversible via rollback but wide blast radius; frequency varies by team.
What this lab is
- A graded, in-browser lab: ten realistic agent actions — a small refund, a $5,000 refund, deleting a prod table, rewording a help-doc, a 100k-recipient marketing blast, rotating a key, merging to main, answering a ticket, wiring a payment, deploying to prod — and you assign each an oversight level.
- You are scored against a defensible rubric that penalises BOTH failure modes: under-oversight (you shipped a disaster that a human should have caught) and over-oversight (you gated a trivial, reversible action until the team stopped using the tool).
- The whole point: human-in-the-loop is a dial, not a switch. The skill being trained is matching oversight to stakes — not maximising oversight.
- Every answer comes with an explanation of why that level fits, and what each wrong direction would have cost in the real world.
- Fully client-side and free — nothing you choose leaves your browser, no account.
Why this matters
The instinct, after a scare, is to put a human in front of everything an agent does. It feels responsible. It is how agents die. Oversight that is spread evenly across trivial and critical actions alike trains reviewers to stop reading, adds latency that makes the tool worse than doing the work by hand, and pushes users to find a way around it — taking the genuinely risky actions out of sight along with the harmless ones. The failure looks like caution right up until the moment it isn't.
Calibration is the actual skill: automate the cheap, reversible, high-frequency actions with a clean audit trail, and spend your scarce human approvals on the irreversible and the high-impact. This lab makes you practise that trade-off ten times and shows you the cost of each miss in both directions, so the judgement transfers to the real system you are building.
Frequently asked questions
What is the Autonomy vs Oversight Lab?
A free, graded exercise that trains the judgement call at the heart of safe AI agents: how much human oversight each action needs. You are shown ten realistic agent actions and you assign each one an oversight level, from fully automating it to requiring two independent human approvals. The lab scores your choices against a defensible rubric and explains both ways an answer can be wrong — too little oversight on a dangerous action, and too much oversight on a trivial one.
Why does the lab penalise too much oversight?
Because over-oversight is a real and common failure, not a safe default. If you put a blocking human approval on rewording a help-doc or answering a routine ticket, the latency and review load make the agent useless, reviewers rubber-stamp everything, and the team routes around the tool — at which point you have lost oversight of the actions that mattered too. Human-in-the-loop is a dial, not a switch. The lab grades you on calibration: matching oversight to the reversibility, blast radius and stakes of each action.
What are the oversight levels?
Five, from most to least autonomous: fully automate (no human in the path); automate with an audit log; async sample review (act now, a human reviews a sample later); blocking human approval (one human must approve before it runs); and two-person dual-control (two independent humans must approve). The rubric places each of the ten actions at the level a well-run team would defend, and explains the cost of being one or more steps off in either direction.
Is this training based on real guidance?
It encodes widely accepted principles — reserve strong, blocking human controls for irreversible, high-blast-radius and safety-, legal- and finance-critical actions; automate reversible, low-stakes, high-frequency actions with good logging. It is an educational lab, not a compliance standard. Your real systems may carry specific regulatory requirements that pin certain decisions to a human regardless of stakes.
This tool is free and always will be. Would a Pro workspace — saved designs, team sharing and exports — be worth paying for?
Show me Pro →No account needed to use this tool — this just tells us whether Pro is worth building out.Educational — general guidance, not security or legal advice.