AI Agent Benchmark Explorer
A curated, cited reference of the benchmarks that evaluate AI agents — systems that take actions through tools, code, the web and real professional work — rather than single-turn chat. Filter by domain, grading method, year and open-vs-proprietary; sort the comparison table; and use the selector to find the 2–3 benchmarks that fit what you are actually measuring. Every row links to its primary source, and numbers that could not be confirmed are shown as “—” rather than invented.
Which benchmark fits my use case?
Pick what you’re evaluating and we’ll suggest 2–3 benchmarks, with why.
Make a selection above to see recommendations.
Explore & compare
The shareable link encodes your filters, sort and pinned selection in the URL — everything stays in your browser.
Showing 16 of 16 benchmarks · sorted by Year ↓.
| Pin to compare | Source | |||||||
|---|---|---|---|---|---|---|---|---|
JobBench Occupational agent work: realistic tasks drawn from a range of professional occupations, scored against detailed per-task rubric chains. Note: License (open vs proprietary) not confirmed from the source. | 2026 | arXiv 2605.26329 | Occupational / professional | 130 across 35 occupations | Rubric Each task is graded by a chain of roughly 35 binary rubric criteria, so credit is partial and tied to specific, checkable sub-steps rather than a single pass/fail. | — | Source → | |
WorkBuddy-Bench Coding agents on real development, PM, QA, ops and security work, evaluated by executing test suites against the agent’s output. Note: Task count not confirmed; shown as "—". | 2026 | Tencent | Software engineering | — | Test suite Coding-agent tasks are verified by running test suites — the agent must produce changes that make the tests pass. | Open | Source → | |
BigLaw Bench: Research Agentic US case-law research: a harder, research-oriented variant of BigLaw Bench built with Snorkel. Note: Task count not confirmed; shown as "—". | 2026 | Harvey + Snorkel | Legal | — | Rubric A harder, agentic extension focused on multi-step US case-law research; rubric-graded like the original BigLaw Bench. | Proprietary | Source → Announced jointly by Harvey and Snorkel; see snorkel.ai. | |
HarnessOpt-Bench A meta-benchmark: rather than scoring an agent on end tasks, it measures LLMs at the task of harness optimization itself. Note: Task count, grading detail and license not confirmed; shown as "—". | 2026 | arXiv 2608.06301 | Agent harness / meta | — | — Evaluates how well an LLM optimises the agent harness itself — the scaffolding, tools and control flow around the model. | — | Source → | |
τ²-bench (tau2-bench) Conversational tool-use agents evaluated in a dual-control setting where the user can also take actions; scored on end state and policy-following rather than a judge’s opinion. Note: Task count not confirmed; shown as "—". | 2025 | Sierra | Workplace tool-use | — | State match A dual-control successor to τ-bench: both the agent and a simulated user can act on the same environment. Success is judged on the final environment/database state together with adherence to domain policies. | Open | Source → arXiv 2506.07982. | |
BrowseComp A browsing benchmark of fact-finding questions whose answers are simple to confirm but require locating information buried across the web. | 2025 | OpenAI | Web navigation | 1266 problems | Exact match Each question has a short, hard-to-find but easy-to-verify answer; grading is exact-match against that gold answer. The difficulty is in the persistent browsing needed to locate the fact, not in checking it. | Open | Source → arXiv 2504.12516. | |
WorkBench Workplace tasks across five tool domains (email, calendar, analytics, CRM, project management) that an agent completes by calling tools against a sandbox. | 2024 | MindsDB / Univ. of Glasgow / Univ. of Warwick | Workplace tool-use | 690 ~690 | State match Outcome-based: the final state of a sandboxed database is compared to the expected state. No LLM judge, so a task passes only if the right end result was produced. | Open | Source → | |
BigLaw Bench Legal work modelled on real billable tasks, graded by rubrics that assess substance and form rather than a single correct answer. Note: Task count not confirmed; shown as "—". | 2024 | Harvey | Legal | — | Rubric Rubrics score both the substance of the legal work and its form, mirroring how a supervising attorney would review a junior’s output. | Proprietary | Source → | |
SWE-bench Verified A human-validated subset of SWE-bench released by OpenAI to remove under-specified or unsolvable tasks; scored by running the project’s own tests on the agent’s code patch. | 2024 | OpenAI | Software engineering | 500 | Test suite A 500-problem subset of SWE-bench, each task vetted by professional software developers so the problem statement and the fail-to-pass tests are confirmed solvable. Grading is the same execution-based check — run the repository’s tests against the agent’s patch. | Open | Source → OpenAI, Aug 2024 — a 500-task curated subset of SWE-bench. | |
AssistantBench Realistic, open-web tasks that require minutes of browsing and multi-step information gathering; scored by comparing the agent’s final answer to a gold answer. | 2024 | Tel Aviv University / Allen Institute for AI | Web navigation | 214 | Exact match Realistic, time-consuming web tasks whose final answer is compared to a gold answer using an answer-type-aware accuracy metric (exact for strings, tolerant for numbers and lists), so credit can be partial rather than strictly all-or-nothing. | Open | Source → arXiv 2407.15711. | |
τ-bench (tau-bench) Tool-using agent in retail and airline customer-service settings, tested against a simulated user and scored on end state and rule-following. Note: Task count not confirmed; shown as "—". | 2024 | Sierra | Workplace tool-use | — | State match A simulated user interacts with the agent; success is judged by the final database state plus adherence to domain rules/policies. | Open | Source → arXiv 2406.12045. | |
MLE-bench Machine-learning engineering tasks drawn from Kaggle competitions, where the agent must train a model and is graded on competition metrics. Note: Task count not confirmed; shown as "—". | 2024 | OpenAI | ML engineering | — | Leaderboard metric Agents tackle Kaggle-style ML engineering competitions; submissions are scored against the competition metric and leaderboard thresholds (e.g. medal lines). | Open | Source → arXiv 2410.07095. | |
SWE-bench Real-world GitHub issues from popular Python repositories; the agent produces a code patch that is scored by whether the project’s tests pass. | 2023 | Princeton | Software engineering | — | Test suite An agent must resolve a real GitHub issue; the patch is validated by running the repository’s own tests (fail-to-pass). | Open | Source → arXiv 2310.06770. | |
GAIA General-assistant questions that require tool use, web browsing and multi-step reasoning, with answers that are simple to verify but hard to produce. Note: Task count not confirmed; shown as "—". | 2023 | Meta / Hugging Face | General assistant | — | Exact match Each question has a single unambiguous answer; grading is exact-match against that gold answer. | Open | Source → arXiv 2311.12983. | |
WebArena Web-navigation tasks on self-hosted clones of real sites (e-commerce, forums, CMS, maps), scored on functional task completion. Note: Task count not confirmed; shown as "—". | 2023 | Carnegie Mellon University | Web navigation | — | State match Functional correctness: the agent operates fully functional, self-hosted web apps and is judged on whether the intended outcome was achieved. | Open | Source → arXiv 2307.13854. | |
LAB (referenced by Harness²) A legal benchmark referred to as "LAB" in the Harness² evaluation (alongside JobBench and WorkBuddy-Bench). We could not independently confirm its definition, authors or details, so every field is left blank. Included only to flag the reference, not to assert specifics. Note: Could not independently confirm the definition of "LAB". All fields left null deliberately. | — | Unconfirmed | — | — | — — | — | Source → Referenced in Harness², arXiv 2610.14224. |
What this tool is
- A free, in-browser reference of real AI AGENT benchmarks — the kind that score an agent on tool use, web actions, code changes and professional work, not single-turn chat Q&A.
- Filter by domain, grading method, year and open vs proprietary; sort a side-by-side comparison table; and use a "which fits my use case?" selector to get 2-3 recommendations with reasons.
- Covers WorkBench, JobBench, WorkBuddy-Bench, BigLaw Bench (+ Research), SWE-bench, SWE-bench Verified, GAIA, τ-bench, τ²-bench, WebArena, BrowseComp, AssistantBench, MLE-bench and HarnessOpt-Bench, with a flagged-as-unconfirmed note on the "LAB" benchmark referenced by Harness².
- Pin up to 4 benchmarks for a focused side-by-side comparison, and copy a shareable link that re-opens your exact filters, sort and pinned set — all encoded in the URL, nothing sent to a server.
- The decision it drills: match the grading method to what you are evaluating — execution/state checks for verifiable work, rubrics for open-ended output, exact-match for single-answer tasks.
- Figures are as reported by each benchmark’s authors; unconfirmed numbers show as "—" rather than a guess. Last reviewed 2026-10-19.
Pick the benchmark that matches the decision
The hardest part of evaluating an agent is not finding a benchmark — it is choosing one whose grading method matches the claim you want to make. If you need a number you can defend, prefer verifiable grading: WorkBench and τ-bench compare the final database state, SWE-bench and WorkBuddy-Bench run the project’s own tests, MLE-bench scores against a competition metric. These leave little room for a judge to be fooled.
For open-ended professional output — legal drafting, occupational tasks — there is no single correct answer, so rubric grading (BigLaw Bench, JobBench) is the honest choice, as long as you treat the rubric itself as part of the measurement. And for questions with one unambiguous answer, exact-match (GAIA) is clean and cheap. The selector above encodes this: tell it your domain, whether you need verifiable grading, and your task type, and it recommends from the same cited dataset — never a benchmark whose details we could not confirm.
How these benchmarks grade — a short glossary
The single most useful thing to know about an agent benchmark is how it decides a task passed. These are the methods that separate the entries above.
- Final-DB-state / environment-state match
- The agent acts on a sandboxed environment (a database, a set of tools), and the benchmark compares the end state of that environment to a reference state. There is no judge in the loop, so a task passes only if the right result was actually produced — which makes the score hard to game but blind to how the agent got there. Used by WorkBench and τ-bench.
- Test-suite / execution grading
- The agent produces code, and the benchmark runs the project’s own tests against it — typically a fail-to-pass check: tests that failed before the change must pass after it. It is objective and reproducible, but only as complete as the test suite, so an under-tested repository can mark a wrong fix as correct. Used by SWE-bench and its human-validated SWE-bench Verified subset.
- Rubric-chain (binary criteria)
- For open-ended work with no single correct output, each task is scored against a chain of specific, checkable criteria — often binary pass/fail sub-steps — so credit is partial and tied to concrete requirements rather than one overall judgement. The quality of the measurement is the quality of the rubric, and the checks are frequently applied by a human or a model. Used by JobBench and BigLaw Bench.
- Exact-match (gold answer)
- The task has a single unambiguous answer, and the agent’s final answer is compared to that gold answer. It is cheap and clean to score, but only works where the answer really is unambiguous; some benchmarks relax strict string-identity with tolerant comparison for numbers and lists. Used by GAIA, BrowseComp and AssistantBench.
- LLM-judge
- A separate language model scores the output, often against a rubric or a reference answer. It scales to open-ended responses that no exact-match check can handle, but it introduces the judge’s own biases — position and verbosity effects, and self-preference — so its agreement with human raters has to be measured, not assumed. None of the benchmarks above relies on it as the primary grader, which is itself worth noting; see Zheng et al., “Judging LLM-as-a-Judge” (arXiv 2306.05685).
Frequently asked questions
What is an AI agent benchmark, and how is it different from a model benchmark?
An agent benchmark scores a full system — a model plus its tools, memory and control loop (its "harness") — on multi-step tasks that require taking actions: calling tools, editing files, navigating web apps or completing professional work. It is graded on outcomes (did the end state match, did the tests pass) rather than on a single answer to a single prompt. A model benchmark, by contrast, usually measures a model in isolation on static questions. The benchmarks in this explorer are all agent benchmarks: WorkBench and τ-bench check the final database state, SWE-bench and WorkBuddy-Bench run test suites, GAIA uses exact-match answers to tool-driven tasks.
Why does the grading method matter so much?
Because the grading method decides what a score actually means. State-match and test-suite grading are verifiable: the task either reached the correct end state or it did not, with no judge in the loop, so the number is hard to game. Rubric grading (BigLaw Bench, JobBench) is right for open-ended professional work where there is no single correct output, but it depends on the rubric’s quality and often on a model or human checker. Exact-match (GAIA) is clean but only works when the answer is unambiguous. When you choose a benchmark, match its grading to your use case rather than chasing the highest headline number — that is exactly what the use-case selector on this page helps with.
Are the task counts and scores on this page verified?
The figures shown are those reported by each benchmark’s own authors, and we deliberately do not invent any. Where a task count or other detail could not be independently confirmed, the field shows "—" instead of a guessed number, and a visible note flags it. One entry, a legal benchmark referred to as "LAB" in the Harness² paper, is included only to record the reference: we could not confirm its definition, so every field is left blank. Always verify against the primary source — each row links to it — before citing a number.
Is this tool free, and does it send my data anywhere?
It is completely free with no signup. The explorer is 100% client-side: the dataset is static and bundled with the page, all filtering, sorting and the use-case selector run in your browser, and nothing you select is sent to any server or API. The only outbound links are to each benchmark’s own source page, which open in a new tab.