Agent Eval Integrity Lab
A queue of agent and LLM evaluations lands on your desk — an executor with open network egress, a score that jumps when web access is on, a memorised public test set, a subagent grading its own output, a leaked train/test split, a canary string turning up in outputs, and an agent reward-hacking a weak grader. For each, name the integrity threat, decide whether the result is trustworthy or compromised, and pick the control that fits — because a score that could reach the rubric or answers is not a measurement. Grounded in Google Research's Harness² (arXiv 2610.14224): restrict the network before evaluation. Real threats, real controls, no invented CVEs.
The real job. More noise, a tighter clock.
Evaluation queue
1 / 9 triaged
executor: unrestricted egress · rubric host public · setup · network
The evaluation executor can reach the open internet during the run, and the benchmark's rubric and reference answers are published on a reachable host.
1 · Integrity threat
2 · Verdict
3 · Control
Pick a threat, the verdict, and a control to continue.
Threat × score-inflation map
This item: name the threat and whether its score could be inflated.
Triage
A queue of agent / LLM evaluation setups and observed behaviours. For each of 9 items, decide the integrity threat (or none), whether the result is trustworthy or compromised, and the control that fits.
So far: 0 / 9 items triaged.
Live arena
4 activeadversary staging…
What this lab is
- A free, browser-based triage lab for agent / LLM evaluation integrity: work a queue of evaluation setups and observed agent behaviours and decide whether each score can be trusted.
- For every item you name the integrity THREAT (rubric/answer leakage, network-to-answers, train/test contamination, memorisation, reward hacking, self-grading bias — or "none, properly isolated"), the VERDICT (trustworthy vs compromised), and the CONTROL that fits.
- Grounded in Google Research's Harness² (arXiv 2610.14224), which warns that executor network access is unrestricted by default and public benchmark rubrics are reachable online — so you must restrict the network before evaluation to keep the agent away from the answers and grading criteria.
- The core skill it drills: a score that could have reached the rubric or answers is compromised (block egress before scoring); contamination and memorisation need held-out isolation and canary/dedup checks; self-grading needs blind independent grading; reward hacking needs a robust rubric — and a properly isolated run must not be over-flagged.
- An evaluation queue plus a live threat × score-inflation risk map, difficulty tiers, a speed bonus and an S–D grade.
Restrict the network before you trust the score
Most agent-evaluation mistakes come from trusting a number without asking whether the agent could have reached the answers. Google Research's Harness² (arXiv 2610.14224) makes the point directly: the evaluation executor's network access is typically unrestricted, and public benchmark rubrics and grading criteria are often reachable online — so you must restrict the network before evaluation to keep the agent away from the answers. A self-improving loop that can read the rubric does not get better at the task; it "improves" by cheating, tailoring its output to the grading criteria. The queue in this lab is built to force those distinctions: rubric / answer leakage and network-to-answers call for blocking egress to the answer and rubric hosts during scoring; contamination and memorisationcall for held-out isolation plus canary strings and de-duplication; self-grading calls for blind, independent grading; and reward hacking calls for a robust, hard-to-game rubric.
Just as important: do not over-flag. A run where egress to the answer and rubric hosts is blocked during scoring, the test set is held-out and private, and an independent blind grader scores against a robust rubric is the trustworthy case — even when the agent legitimately used the web on a scoped allowlist. The whole skill of eval-integrity triage is telling the run that could have read the answers apart from the one that provably could not, and spending your scrutiny where a score is actually inflated.
Frequently asked questions
What does the Agent Eval Integrity Lab teach?
It teaches how to triage whether an agent or LLM evaluation result can be trusted. Given an evaluation setup or an observed behaviour — an executor with open network egress, a score that jumps when web access is enabled, a memorised public test set, a subagent grading its own output, a leaked train/test split, a canary string appearing in outputs, or an agent gaming a weak grader — you decide the integrity threat, whether the result is trustworthy or compromised, and the control that fits. It makes the key distinction explicit: a score that could have reached the answers or rubric is not a measurement of capability.
What is Harness² and why does it matter here?
Harness² (Google Research, arXiv 2610.14224) is a study of agent-evaluation harness security. It warns that the evaluation executor's network access is typically unrestricted and that public benchmark rubrics and grading criteria may be reachable online, so you must restrict the network before evaluation to prevent the agent from accessing the answers or grading criteria. It also notes that a self-improving loop which can read the rubric can "improve" by cheating — tailoring output to the grading criteria rather than getting better at the task. This lab drills exactly that discipline: isolate the network before you trust a number.
Why is a properly isolated run treated as trustworthy even when the agent had network access?
Because network access is not automatically disqualifying — what matters is whether the agent could reach the answers or rubric. An evaluation where egress to the answer and rubric hosts is blocked during scoring, the test set is held-out and private (de-duplicated against training data), and grading is independent and blind against a robust rubric is the trustworthy case, even for a browsing agent on an allowlist. The skill of triage is telling that run apart from one where the executor could read the grading criteria, and not over-flagging the isolated one.
Is this a real evaluation tool or does it touch my models or benchmarks?
No. It is an educational, offline triage exercise with synthetic scenarios. It does not run any evaluation, connect to any model, benchmark or grader, and contains no exploit detail — only the defender decision of threat, verdict and control. It is meant to build the judgement an eval engineer or ML/security owner uses to decide whether a reported agent or LLM score is sound or inflated. The threats and controls are grounded in Harness² and standard benchmark-hygiene practice, with no invented CVE numbers.