Self-Improving Agent Audit Lab
A self-improving agent harness — the kind described in Google Research's Harness² work (arXiv 2610.14224) — edits its own harness and carries forward lessons over time. It is in production. Your job is to audit it: for each observation, decide the control area it belongs to, give a verdict, and pick the remediation or evidence that closes it — because an unrecorded edit, a rewritable log, an untested rollback, an ungoverned change, unwatched drift and an agent widening its own tools each call for a different fix, and a fully evidenced, reversible, well-owned control should be left alone. Mapped to ISO/IEC 42001 and ISO/IEC 27090 by theme, with no quoted clause numbers and no invented CVEs.
The real job. More noise, a tighter clock.
Audit register
1 / 9 reviewed
OBS-01 · harness changed 3× this quarter · adopted edits: present · adoption record: none
The agent has rewritten parts of its own harness three times this quarter. There is no record of which edits were adopted, from what source, or whether the running harness matches any reviewed version.
1 · Audit control area
2 · Verdict
3 · Remediation / evidence that closes it
Pick a control area, a verdict, and the remediation to continue.
Control-area × gap-severity map
This observation: place its control area and decide how big the gap is.
Audit
A deployed self-improving agent that edits its own harness and persists lessons. For each of 9 observations, place the audit control area, give a verdict, and pick the remediation or evidence that closes it.
So far: 0 / 9 observations assessed.
Live arena
4 activeadversary staging…
What this lab is
- A free, browser-based audit lab: you are the AUDITOR of a deployed self-improving agent harness (Harness², Google Research, arXiv 2610.14224) — one that edits its own harness and persists "lessons" over time. The drill: can you prove what it changed, why, and roll it back?
- For every observation you pick the AUDIT CONTROL AREA (provenance of adopted edits, decision/evidence logging, rollback capability, ownership/accountability, drift detection, or non-self-expansion of scope), a VERDICT (conformant / opportunity / minor gap / major gap), and the remediation or evidence that closes it.
- Control expectations are tied to ISO/IEC 42001 (AI management system: lifecycle, logging, management review, change control) and ISO/IEC 27090 (security guidance for AI systems) — described by theme, with no clause numbers quoted verbatim and no invented CVEs.
- The core skill it drills: an unrecorded edit needs signed provenance, an agent-writable log needs an immutable adoption record, snapshots never restored need a tested rollback, ungoverned changes need a named owner + approval gate, unwatched behaviour needs drift alerts, and an agent widening its own tools must be capped — and a fully evidenced, reversible, well-owned control must NOT be over-flagged.
- An audit register plus a live control-area × gap-severity map, difficulty tiers, a speed bonus and an S–D grade.
Prove it, or roll it back
A self-improving agent is useful precisely because it changes itself — and that is also what makes it hard to govern. The observations in this lab are built to force the distinctions that matter. Provenance is being able to say which edits the agent adopted, from where, and that they are authentic — a signed, traceable record per change. Logging is capturing the decision and the evidence behind each adopted edit, in a record the agent itself cannot rewrite. Rollback is a tested path back to a known-good harness — snapshots you have actually restored, not just kept. And non-self-expansion is the rule that the agent must never grant itself new tools or scope; a permission change is exactly what needs a human approval gate. These map to the lifecycle, logging, management-review and change-control expectations of ISO/IEC 42001 and the AI-security guidance of ISO/IEC 27090.
Just as important: do not over-flag. An agent whose edits are logged append-only, whose rollback is tested and passing, whose scope changes are all gated and denied by default, and which has a named accountable owner is the conformant case — the right call is to keep those controls running, not to tear the agent out. The whole skill of this audit is telling the unrecorded, irreversible, self-expanding agent apart from the one that is already doing the right things, and spending your limited remediation effort where it moves risk the most.
Frequently asked questions
What is a self-improving agent harness, and what does this lab teach?
A self-improving agent harness is an agent that can edit its own harness — the scaffolding of prompts, tools and routines around the model — and carry forward "lessons" across sessions, as described in Google Research's Harness² work (arXiv 2610.14224). That is powerful and also an assurance problem: the thing you deployed keeps changing itself. This lab is the auditor drill for exactly that situation. For each observation about a deployed self-improving agent you decide which audit control area it belongs to, whether it is conformant or a gap, and the remediation or evidence that would close it — so you can prove what the agent changed, why it kept the change, and that you can roll it back.
Which standards does it map to?
It maps the controls to ISO/IEC 42001, the AI management system standard — its expectations around the AI lifecycle, logging and record-keeping, management responsibility and review, and change control — and to ISO/IEC 27090, which gives guidance on addressing security threats to AI systems (for example, an agent silently widening its own scope). The lab references these by theme and never quotes clause numbers verbatim or invents them; the goal is to recognise the control area and choose the fix, not to memorise a citation.
Why is "the agent widened its own tool scope" treated as more than a bug?
Because a self-improving agent that can grant itself new tools or privileges can expand what it is able to do without anyone deciding to let it — and no single patch fixes that, it is a design property. The remediation is to cap self-expansion: the agent must not be able to change its own scope, and any scope or permission change must route through an explicit human approval gate that denies by default. ISO/IEC 27090 security guidance treats uncontrolled self-expansion as a threat to address, not a capability to welcome. The same logic is why the adoption log must sit outside the agent's own write scope: a record the subject can rewrite is not evidence.
Is this a real scanner, or does it touch a live agent?
No. It is an educational, offline audit exercise with a synthetic set of observations about a fictional deployed agent. It does not connect to, scan, or modify any agent, model or system, and it contains no exploit detail — only the auditor's decision of control area, verdict and remediation. It is meant to build the judgement an AI-governance, GRC or assurance owner uses when a self-improving agent lands in their scope: separating a genuine, evidenced, reversible control from an unrecorded, irreversible, self-expanding one — and not over-flagging the conformant case.