Agent Benchmarks, Compared: WorkBench, JobBench, BigLaw Bench, SWE-bench, GAIA, τ-bench and the Rest — What a CISO Should Actually Read
AI-agent benchmarks measure very different jobs — writing code, researching case law, booking a flight through a tool, finding a hard fact on the web — so a strong score on one tells you little about another. They also grade differently: some check a machine-verifiable outcome (programmatic), some score against an expert rubric, and some lean on human or LLM judgement, which changes how much a number can be trusted and reproduced. A score is always the model plus its harness, not the model alone, which is why Google’s Harness² work (which reports gains from optimising the scaffolding around a fixed model) matters to anyone reading a leaderboard. For a leader, the move is to match the benchmark to your use case, ask how it was graded, and treat any single number as the start of a question, not an answer.
Every agent vendor now arrives with a benchmark number. "Beats the field on SWE-bench." "State of the art on GAIA." "Top of BigLaw Bench." The numbers are not meaningless — but a strong score on one of these says almost nothing about the others, and the grading method underneath each one decides how much the number is worth at all. This is a security-leadership field guide to the agent benchmarks you are most likely to be shown: what each measures, how it grades, and the questions to ask before a number earns a place in a decision.
Analysis for CISOs, AI-platform and GRC teams. Every benchmark below is linked to its primary source. Where we are not certain a name maps to a real, published benchmark, we say so explicitly rather than describe it as established — the whole point of this piece is to read benchmarks carefully.
If you would rather poke at this interactively, our Agent Benchmark Explorer is the companion to this post — filter the benchmarks by task type and grading method and see which ones actually resemble your use case.
The one idea to hold onto: a score is the whole system, not the model
Before any single benchmark, internalise this: an agent’s benchmark score is a property of the model plus its harness — the prompts, tools, memory, orchestration and guardrails wrapped around the model — run against a specific task set and measured by a specific grader. Change any of the three and the number moves. Google Cloud AI Research and Northwestern University’s Harness² work (arXiv 2610.14224) makes this concrete: it optimises the harness around a fixed model and reports meaningful gains on several professional-work benchmarks — the model never changed. Harness² is a method; the benchmarks are the yardsticks it is measured against. So "model X scores Y" is an incomplete sentence. The complete one is "model X, in this harness, on this benchmark split, graded this way, scored Y."
How benchmarks grade — the distinction that matters most
There are three broad grading styles, and the style tells you how much a number can be trusted and reproduced.
- Programmatic / outcome-based. A machine checks a verifiable fact: did the repository’s test suite pass, does the database now match the goal state, does the short answer equal the key. Reproducible and hard to fudge — but only works where "correct" is unambiguous. (SWE-bench, GAIA, τ-bench, WebArena, MLE-bench, BrowseComp, AssistantBench, WorkBench.)
- Rubric-based. An open-ended output is scored against a checklist — often expert-authored or fact-anchored — awarding points for required elements and penalising errors. Captures quality a yes/no check cannot, but introduces judgement and is harder to reproduce exactly. (BigLaw Bench, JobBench.)
- Human or LLM-as-judge. The softest signal: nuanced, but the most expensive, least reproducible, and most sensitive to who or what is judging. Rubric benchmarks often lean on this at the edges.
As a rule of thumb: a programmatic outcome score is the most defensible, a rubric score needs the rubric disclosed before you trust it, and a bare "LLM-judged" score with no rubric is a vibe, not a measurement.
The benchmarks, one by one
WorkBench — workplace tool use, graded by outcome
WorkBench (Styles et al., COLM 2024; code) puts an agent in a sandboxed office: five databases, 26 tools, 690 tasks like sending email, scheduling meetings and updating records. Its signature is outcome-centric grading — each task has a unique, unambiguous correct end-state, so the grader checks the final database state rather than the agent’s steps, which means any route to the right outcome counts. The original paper found baseline agents weak (the best configuration resolved well under half of tasks; weaker models far less) and noted a safety-relevant failure mode: wrong actions, like an email sent to the wrong person. For a leader, WorkBench is the closest public proxy for "can this agent operate our internal tools without doing damage."
JobBench — delegable professional work, graded by rubric
JobBench ("Aligning Agent Work With Human Will," 2026) covers 130 agentic tasks across roughly 35 occupations, each delivered as a cluttered workspace of mixed reference files so the agent must sort signal from noise the way a real job does. Grading is a fact-anchored chain of rubrics — on the order of three dozen binary criteria per task — so it scores work quality, not just a single end-state. Reported top scores sit well below human reliability (the paper’s best configuration is under half of criteria met). Treat JobBench as a read on open-ended knowledge work, and remember the rubric is doing the judging.
BigLaw Bench (+ BigLaw Bench: Research) — legal work-product, expert rubrics
BigLaw Bench (Harvey, 2024) scores legal outputs against custom, lawyer-authored rubrics covering answer quality and source reliability, with positive points for required elements and negative points for errors such as hallucinated citations. BigLaw Bench: Research (2026, built with Snorkel AI) extends it to hard agentic legal research — can a model search, reason over and cite case law end to end like a practising lawyer. Vendor-published scores (Harvey’s own figures, not independent measurements) show frontier models improving over time; treat them as directional and vendor-sourced. This is a useful reminder that in high-stakes professional domains, the grader is a domain expert’s rubric, and the rubric is the benchmark.
SWE-bench / SWE-bench Verified — real software fixes, programmatic
SWE-bench (Jimenez et al.) gives an agent a real GitHub issue and repository and asks for a patch; grading is purely programmatic — apply the patch, run the repo’s test suite, pass or fail. SWE-bench Verified is a 500-instance, human-reviewed subset (released with OpenAI’s involvement) that removes under-specified or unsolvable problems so the number means more. One caveat worth verifying before you quote it: there is reporting that Verified has drawn contamination and shortcut concerns and that newer variants (e.g. "SWE-bench Pro") are emerging — check the current state of the benchmark family rather than assuming a 2024 number still stands. SWE-bench is the benchmark to look at for autonomous coding agents, and its programmatic grading makes it one of the harder ones to fake.
GAIA — general assistant reasoning, exact-answer
GAIA (Mialon et al., Meta/Hugging Face) is 466 hand-curated questions that need reasoning, tool use, web browsing and sometimes multimodality, across three difficulty levels. Answers are short and factual and graded by quasi-exact match against a held-out key — easy to verify, and resistant to waffle. GAIA is a good read on "general assistant competence," but note that a held-out leaderboard invites overfitting over time, so a very high GAIA score should prompt the contamination question.
τ-bench / τ²-bench — customer-service tool use, outcome + reliability
τ-bench (Sierra) has an agent hold a realistic conversation with a simulated user while calling domain APIs (retail, airline) and following a written policy; grading compares the final database state to the goal, and it reports pass^k — success across k repeated trials — which exposes reliability, not just best-case luck. τ²-bench (Barres et al., ICML 2026) extends this to a dual-control telecom domain where both the agent and the user can act, modelled as a Dec-POMDP, and shows performance dropping sharply once the agent must guide a user rather than act alone. For anyone deploying customer-facing agents, τ-bench’s pass^k is the single most production-relevant signal on this list: it measures whether the agent is consistently right.
WebArena — autonomous web navigation, functional checks
WebArena (Zhou et al., CMU) runs agents against self-hosted, reproducible websites — e-commerce, a forum, a code host, a CMS, maps — with 812 tasks graded by functional correctness (did the intended state change happen, or does the retrieved information match). Because the sites are self-hosted, runs are reproducible in a way the live web never is. WebArena is the read on "can this agent drive a browser through a multi-step task," a capability with obvious data-handling and blast-radius implications.
MLE-bench — machine-learning engineering, medal thresholds
MLE-bench (OpenAI, ICLR 2025) sets agents loose on 75 real Kaggle competitions — prepare data, train models, run experiments — and grades programmatically against Kaggle’s own medal thresholds using the public leaderboards as human baselines. The headline from the paper (best setup reaching bronze-medal level in a modest fraction of competitions) is a reminder that "can build ML systems autonomously" is still hard. Relevant if a vendor claims an agent can do data-science or ML work unattended.
BrowseComp — hard-to-find facts, verifiable answers
BrowseComp (OpenAI, 2025) is 1,266 questions deliberately engineered so the answer is hard to find but easy to verify — the agent must browse persistently, filter noise and synthesise, then produce a short answer checked against a key. It isolates deep, multi-hop web research. A strong BrowseComp score says an agent is good at finding buried facts; it says nothing about whether it can safely act on them.
AssistantBench — realistic, time-consuming web tasks
AssistantBench (Yoran et al., EMNLP 2024) is 214 realistic, open-web tasks that take real time — monitoring a market, finding suitable nearby businesses — with automatic answer verification and a metric that gives partial credit. Scores at launch were low (no model much above the mid-20s percent), which is itself the signal: realistic open-web assistance is still unsolved. Useful as a reality check against polished demos.
Two names to handle with care: WorkBuddy-Bench and LAB
The Harness² paper (arXiv 2610.14224) reports its results on three benchmarks: WorkBuddy-Bench, JobBench and a benchmark called "LAB." JobBench has its own primary source (arXiv 2605.26329); for WorkBuddy-Bench and LAB we could not find an independent, standalone primary source outside the Harness² paper itself, so we do not treat them as established, independently-published benchmarks the way we do the others on this list. If a vendor cites either, ask for the primary source — and whether the number came from the Harness² evaluation or somewhere else — before you accept it. This is exactly the habit the rest of this post is arguing for.
What a security or ops leader should actually do with a benchmark number
- Match the benchmark to your use case. Coding (SWE-bench), customer-service tool use (τ-bench), web navigation (WebArena), deep research (BrowseComp/GAIA), professional knowledge work (JobBench/BigLaw Bench), ML engineering (MLE-bench) and office tool use (WorkBench) are different jobs. A score off-target is noise.
- Ask how it was graded. Programmatic outcome, expert rubric, or LLM/human judge — and if rubric or judge, is the rubric disclosed? An undisclosed-rubric or bare LLM-judge number is not evidence.
- Pin the split and version. SWE-bench vs Verified vs Pro; original GAIA vs a later revision. Was contamination controlled and were test answers held out?
- Prefer reliability over best-case. pass^k (τ-bench) over pass@1. For anything customer-facing, consistency is the number that matters.
- Make them disclose the harness. Scaffolding, tools and prompts move scores by double digits — Harness² is the proof. "Which harness produced this?" is a fair and revealing question.
- Date it. Benchmarks age, leak and get gamed. A two-year-old leaderboard position is a historical note, not a current fact.
Where this plugs into the governance you already run
Benchmarks are a procurement and assurance input, not a control. The capability a benchmark measures becomes your risk the moment the agent touches your systems — so pair the score with a threat model (LLM Threat-Model Generator), a provenance record of the model and its components (AIBOM generator), and the harness-governance questions a self-improving agent raises (Harness Designer). A benchmark tells you what an agent can do on a test set; your governance decides what it is allowed to do in production.
The honest takeaway
No single benchmark certifies an agent, and cross-benchmark comparisons are mostly apples-to-oranges. Read three things every time: what job the benchmark measures, how it grades, and what harness produced the score. Do that and a leaderboard becomes a useful instrument. Skip it and a benchmark number is just a vendor’s adjective with a decimal point.
Sources & credit
- WorkBench — arXiv 2405.00823 (Styles et al., COLM 2024); code.
- JobBench — arXiv 2605.26329 ("Aligning Agent Work With Human Will").
- BigLaw Bench & BigLaw Bench: Research — Harvey: Introducing BigLaw Bench, Expanding BigLaw Bench.
- SWE-bench — arXiv 2310.06770 (Jimenez et al.); SWE-bench Verified is the human-reviewed 500-instance subset (verify current status of the family).
- GAIA — arXiv 2311.12983 (Mialon et al., Meta/Hugging Face).
- τ-bench — arXiv 2406.12045 (Sierra); τ²-bench — ICML 2026 (Barres et al.).
- WebArena — arXiv 2307.13854 (Zhou et al., CMU).
- MLE-bench — arXiv 2410.07095 (OpenAI, ICLR 2025).
- BrowseComp — arXiv 2504.12516 (OpenAI).
- AssistantBench — arXiv 2407.15711 (Yoran et al., EMNLP 2024).
- Harness² — arXiv 2610.14224 (Google Cloud AI Research and Northwestern University); verify specific score figures against the primary source before quoting them.
Published 19 Oct 2026. Where this summary and a benchmark’s primary source differ, the primary source is authoritative. WorkBuddy-Bench and LAB are the Harness² paper’s own evaluation benchmarks; we found no independent primary source for them outside that paper as of publication.
Ready to practise the decisions these articles describe?
Run a free War Room →