# Agent Benchmarks, Compared: WorkBench, JobBench, BigLaw Bench, SWE-bench, GAIA, τ-bench and the Rest — What a CISO Should Actually Read

> A security-leadership field guide to the AI-agent benchmarks vendors quote: WorkBench, JobBench, BigLaw Bench (+ BigLaw Bench: Research), SWE-bench / SWE-bench Verified, GAIA, τ-bench / τ²-bench, WebArena, MLE-bench, BrowseComp and AssistantBench. What each one measures, how grading differs (programmatic vs rubric vs human), why a high score on one says almost nothing about another, and the questions to ask when a benchmark number lands on your desk. Tied to Google’s Harness² work, which is a reminder that a score is the model plus its harness — not the model alone.

Source: https://playciso.com/blog/agent-benchmarks-compared-workbench-jobbench-biglaw-bench · Published: 2026-10-19 · Publisher: PlayCISO (https://playciso.com)

---

Every agent vendor now arrives with a benchmark number. "Beats the field on SWE-bench." "State of the art on GAIA." "Top of BigLaw Bench." The numbers are not meaningless — but a strong score on one of these says almost nothing about the others, and the grading method underneath each one decides how much the number is worth at all. This is a security-leadership field guide to the agent benchmarks you are most likely to be shown: what each measures, how it grades, and the questions to ask before a number earns a place in a decision. _Analysis for CISOs, AI-platform and GRC teams. Every benchmark below is linked to its primary source. Where we are not certain a name maps to a real, published benchmark, we say so explicitly rather than describe it as established — the whole point of this piece is to read benchmarks carefully._ If you would rather poke at this interactively, our [Agent Benchmark Explorer](/tools/agent-benchmark-explorer) is the companion to this post — filter the benchmarks by task type and grading method and see which ones actually resemble your use case. ## The one idea to hold onto: a score is the whole system, not the model Before any single benchmark, internalise this: an agent’s benchmark score is a property of the **model plus its harness** — the prompts, tools, memory, orchestration and guardrails wrapped around the model — run against a **specific task set** and measured by a **specific grader**. Change any of the three and the number moves. Google Cloud AI Research and Northwestern University’s **Harness²** work ([arXiv 2610.14224](https://arxiv.org/abs/2610.14224)) makes this concrete: it _optimises the harness around a fixed model_ and reports meaningful gains on several professional-work benchmarks — the model never changed. Harness² is a method; the benchmarks are the yardsticks it is measured against. So "model X scores Y" is an incomplete sentence. The complete one is "model X, in this harness, on this benchmark split, graded this way, scored Y." ## How benchmarks grade — the distinction that matters most There are three broad grading styles, and the style tells you how much a number can be trusted and reproduced. - Programmatic / outcome-based. A machine checks a verifiable fact: did the repository’s test suite pass, does the database now match the goal state, does the short answer equal the key. Reproducible and hard to fudge — but only works where "correct" is unambiguous. (SWE-bench, GAIA, τ-bench, WebArena, MLE-bench, BrowseComp, AssistantBench, WorkBench.) - Rubric-based. An open-ended output is scored against a checklist — often expert-authored or fact-anchored — awarding points for required elements and penalising errors. Captures quality a yes/no check cannot, but introduces judgement and is harder to reproduce exactly. (BigLaw Bench, JobBench.) - Human or LLM-as-judge. The softest signal: nuanced, but the most expensive, least reproducible, and most sensitive to who or what is judging. Rubric benchmarks often lean on this at the edges. As a rule of thumb: a programmatic outcome score is the most defensible, a rubric score needs the rubric disclosed before you trust it, and a bare "LLM-judged" score with no rubric is a vibe, not a measurement. ## The benchmarks, one by one ### WorkBench — workplace tool use, graded by outcome [WorkBench](https://arxiv.org/abs/2405.00823) (Styles et al., COLM 2024; [code](https://github.com/olly-styles/WorkBench)) puts an agent in a sandboxed office: five databases, 26 tools, 690 tasks like sending email, scheduling meetings and updating records. Its signature is **outcome-centric grading** — each task has a unique, unambiguous correct end-state, so the grader checks the final database state rather than the agent’s steps, which means any route to the right outcome counts. The original paper found baseline agents weak (the best configuration resolved well under half of tasks; weaker models far less) and noted a safety-relevant failure mode: wrong actions, like an email sent to the wrong person. For a leader, WorkBench is the closest public proxy for "can this agent operate our internal tools without doing damage." ### JobBench — delegable professional work, graded by rubric [JobBench](https://arxiv.org/abs/2605.26329) ("Aligning Agent Work With Human Will," 2026) covers 130 agentic tasks across roughly 35 occupations, each delivered as a cluttered workspace of mixed reference files so the agent must sort signal from noise the way a real job does. Grading is a **fact-anchored chain of rubrics** — on the order of three dozen binary criteria per task — so it scores work quality, not just a single end-state. Reported top scores sit well below human reliability (the paper’s best configuration is under half of criteria met). Treat JobBench as a read on open-ended knowledge work, and remember the rubric is doing the judging. ### BigLaw Bench (+ BigLaw Bench: Research) — legal work-product, expert rubrics [BigLaw Bench](https://www.harvey.ai/blog/introducing-biglaw-bench) (Harvey, 2024) scores legal outputs against **custom, lawyer-authored rubrics** covering answer quality and source reliability, with positive points for required elements and negative points for errors such as hallucinated citations. [BigLaw Bench: Research](https://www.harvey.ai/blog/expanding-big-law-bench) (2026, built with Snorkel AI) extends it to hard agentic legal research — can a model search, reason over and cite case law end to end like a practising lawyer. Vendor-published scores (Harvey’s own figures, not independent measurements) show frontier models improving over time; treat them as directional and vendor-sourced. This is a useful reminder that in high-stakes professional domains, the grader is a domain expert’s rubric, and the rubric is the benchmark. ### SWE-bench / SWE-bench Verified — real software fixes, programmatic [SWE-bench](https://arxiv.org/abs/2310.06770) (Jimenez et al.) gives an agent a real GitHub issue and repository and asks for a patch; grading is **purely programmatic** — apply the patch, run the repo’s test suite, pass or fail. **SWE-bench Verified** is a 500-instance, human-reviewed subset (released with OpenAI’s involvement) that removes under-specified or unsolvable problems so the number means more. One caveat worth verifying before you quote it: there is reporting that Verified has drawn contamination and shortcut concerns and that newer variants (e.g. "SWE-bench Pro") are emerging — check the current state of the benchmark family rather than assuming a 2024 number still stands. SWE-bench is the benchmark to look at for autonomous coding agents, and its programmatic grading makes it one of the harder ones to fake. ### GAIA — general assistant reasoning, exact-answer [GAIA](https://arxiv.org/abs/2311.12983) (Mialon et al., Meta/Hugging Face) is 466 hand-curated questions that need reasoning, tool use, web browsing and sometimes multimodality, across three difficulty levels. Answers are short and factual and graded by **quasi-exact match against a held-out key** — easy to verify, and resistant to waffle. GAIA is a good read on "general assistant competence," but note that a held-out leaderboard invites overfitting over time, so a very high GAIA score should prompt the contamination question. ### τ-bench / τ²-bench — customer-service tool use, outcome + reliability [τ-bench](https://arxiv.org/abs/2406.12045) (Sierra) has an agent hold a realistic conversation with a simulated user while calling domain APIs (retail, airline) and following a written policy; grading **compares the final database state to the goal**, and it reports **pass^k** — success across k repeated trials — which exposes reliability, not just best-case luck. [τ²-bench](https://icml.cc/virtual/2026/oral/71171) (Barres et al., ICML 2026) extends this to a _dual-control_ telecom domain where both the agent _and_ the user can act, modelled as a Dec-POMDP, and shows performance dropping sharply once the agent must guide a user rather than act alone. For anyone deploying customer-facing agents, τ-bench’s pass^k is the single most production-relevant signal on this list: it measures whether the agent is _consistently_ right. ### WebArena — autonomous web navigation, functional checks [WebArena](https://arxiv.org/abs/2307.13854) (Zhou et al., CMU) runs agents against **self-hosted, reproducible websites** — e-commerce, a forum, a code host, a CMS, maps — with 812 tasks graded by **functional correctness** (did the intended state change happen, or does the retrieved information match). Because the sites are self-hosted, runs are reproducible in a way the live web never is. WebArena is the read on "can this agent drive a browser through a multi-step task," a capability with obvious data-handling and blast-radius implications. ### MLE-bench — machine-learning engineering, medal thresholds [MLE-bench](https://arxiv.org/abs/2410.07095) (OpenAI, ICLR 2025) sets agents loose on 75 real Kaggle competitions — prepare data, train models, run experiments — and grades **programmatically against Kaggle’s own medal thresholds** using the public leaderboards as human baselines. The headline from the paper (best setup reaching bronze-medal level in a modest fraction of competitions) is a reminder that "can build ML systems autonomously" is still hard. Relevant if a vendor claims an agent can do data-science or ML work unattended. ### BrowseComp — hard-to-find facts, verifiable answers [BrowseComp](https://arxiv.org/abs/2504.12516) (OpenAI, 2025) is 1,266 questions deliberately engineered so the answer is **hard to find but easy to verify** — the agent must browse persistently, filter noise and synthesise, then produce a short answer checked against a key. It isolates deep, multi-hop web research. A strong BrowseComp score says an agent is good at finding buried facts; it says nothing about whether it can safely act on them. ### AssistantBench — realistic, time-consuming web tasks [AssistantBench](https://arxiv.org/abs/2407.15711) (Yoran et al., EMNLP 2024) is 214 realistic, open-web tasks that take real time — monitoring a market, finding suitable nearby businesses — with **automatic answer verification** and a metric that gives partial credit. Scores at launch were low (no model much above the mid-20s percent), which is itself the signal: realistic open-web assistance is still unsolved. Useful as a reality check against polished demos. Two names to handle with care: WorkBuddy-Bench and LAB The Harness² paper (arXiv 2610.14224) reports its results on three benchmarks: **WorkBuddy-Bench**, **JobBench** and a benchmark called **"LAB."** JobBench has its own primary source (arXiv 2605.26329); for WorkBuddy-Bench and LAB we could not find an independent, standalone primary source outside the Harness² paper itself, so we do not treat them as established, independently-published benchmarks the way we do the others on this list. If a vendor cites either, ask for the primary source — and whether the number came from the Harness² evaluation or somewhere else — before you accept it. This is exactly the habit the rest of this post is arguing for. ## What a security or ops leader should actually do with a benchmark number - Match the benchmark to your use case. Coding (SWE-bench), customer-service tool use (τ-bench), web navigation (WebArena), deep research (BrowseComp/GAIA), professional knowledge work (JobBench/BigLaw Bench), ML engineering (MLE-bench) and office tool use (WorkBench) are different jobs. A score off-target is noise. - Ask how it was graded. Programmatic outcome, expert rubric, or LLM/human judge — and if rubric or judge, is the rubric disclosed? An undisclosed-rubric or bare LLM-judge number is not evidence. - Pin the split and version. SWE-bench vs Verified vs Pro; original GAIA vs a later revision. Was contamination controlled and were test answers held out? - Prefer reliability over best-case. pass^k (τ-bench) over pass@1. For anything customer-facing, consistency is the number that matters. - Make them disclose the harness. Scaffolding, tools and prompts move scores by double digits — Harness² is the proof. "Which harness produced this?" is a fair and revealing question. - Date it. Benchmarks age, leak and get gamed. A two-year-old leaderboard position is a historical note, not a current fact. ## Where this plugs into the governance you already run Benchmarks are a procurement and assurance input, not a control. The capability a benchmark measures becomes _your_ risk the moment the agent touches your systems — so pair the score with a threat model ([LLM Threat-Model Generator](/tools/llm-threat-model)), a provenance record of the model and its components ([AIBOM generator](/tools/aibom)), and the harness-governance questions a self-improving agent raises ([Harness Designer](/tools/harness-designer)). A benchmark tells you what an agent _can_ do on a test set; your governance decides what it is _allowed_ to do in production. ## The honest takeaway No single benchmark certifies an agent, and cross-benchmark comparisons are mostly apples-to-oranges. Read three things every time: **what job the benchmark measures, how it grades, and what harness produced the score.** Do that and a leaderboard becomes a useful instrument. Skip it and a benchmark number is just a vendor’s adjective with a decimal point. ## Sources & credit - WorkBench — arXiv 2405.00823 (Styles et al., COLM 2024); code. - JobBench — arXiv 2605.26329 ("Aligning Agent Work With Human Will"). - BigLaw Bench & BigLaw Bench: Research — Harvey: Introducing BigLaw Bench, Expanding BigLaw Bench. - SWE-bench — arXiv 2310.06770 (Jimenez et al.); SWE-bench Verified is the human-reviewed 500-instance subset (verify current status of the family). - GAIA — arXiv 2311.12983 (Mialon et al., Meta/Hugging Face). - τ-bench — arXiv 2406.12045 (Sierra); τ²-bench — ICML 2026 (Barres et al.). - WebArena — arXiv 2307.13854 (Zhou et al., CMU). - MLE-bench — arXiv 2410.07095 (OpenAI, ICLR 2025). - BrowseComp — arXiv 2504.12516 (OpenAI). - AssistantBench — arXiv 2407.15711 (Yoran et al., EMNLP 2024). - Harness² — arXiv 2610.14224 (Google Cloud AI Research and Northwestern University); verify specific score figures against the primary source before quoting them. _Published 19 Oct 2026. Where this summary and a benchmark’s primary source differ, the primary source is authoritative. WorkBuddy-Bench and LAB are the Harness² paper’s own evaluation benchmarks; we found no independent primary source for them outside that paper as of publication._