How to Evaluate an AI Agent
Agent leaderboards are everywhere, and almost all of them hand you a single number. This guide shows you how to actually evaluate an agent — define what "good" means for your task, pick the benchmark that matches it, read the grading method honestly, and build a defensible rubric instead of trusting a vendor's headline score.
In plain words
Evaluating an AI agent means deciding what success is for your task, then choosing a benchmark whose tasks look like yours — SWE-bench for coding, tau-bench for tool use, WebArena for the web, GAIA or BrowseComp for research, WorkBench for office work. A high score on a benchmark that does not match your task is noise.
Then read how it is graded (programmatic beats rubric beats judge), prefer reliability (pass^k) over a lucky best-of, and remember the score is the model plus its harness. Finish by building your own weighted rubric and re-testing on your own data before production. PlayCISO has a free benchmark explorer to help you match the right one.
Step by step
- 1
Define what "good" means for your task
Before you look at a single leaderboard, write down what success is for the job you actually have. Is it a correct code patch that passes tests, a tool call with the right arguments, a research answer with real citations, or a completed multi-step office workflow? Pin down the inputs, the acceptable outputs, the failure modes you cannot tolerate, and how you would grade one run by hand. Everything downstream — which benchmark, which grading method, which weight — follows from this. An agent that is excellent at a task shaped differently from yours is not excellent for you.
- 2
Match a benchmark to that task
Benchmarks are task-specific, so pick the one whose shape matches yours. Coding and bug-fixing map to SWE-bench (resolve real GitHub issues against a repo). Tool and API use maps to tau-bench (follow policy while calling tools in a simulated customer-service domain). Web navigation maps to WebArena (complete goals across realistic self-hosted sites). Open-ended research and browsing map to GAIA and BrowseComp (multi-step questions that need real retrieval). Office and knowledge-work automation maps to WorkBench (database-backed tasks like email, calendar and CRM actions). A high score on a benchmark that is off-target for your use case is noise, not evidence.
- 3
Understand the grading method and how far to trust it
How a benchmark scores matters as much as what it tests. Programmatic grading (unit tests pass, the database ends in the expected state, the correct tool was called) is the most trustworthy because it is deterministic and reproducible. Rubric grading scores outputs against a written checklist of required elements and penalized errors — stronger in subjective domains, but only as good as the rubric. Human or LLM-as-judge grading scales to open-ended tasks but introduces grader bias, prompt sensitivity and inconsistency; an LLM judge can be gamed by verbose or confident-sounding answers. Know which method produced a number before you compare numbers.
- 4
Prefer reliability over a lucky best-of
A single top-line score is usually a best-of or an average that hides variance. For anything you will run in production, reliability matters more than a peak: look at pass^k (the probability the agent succeeds on all k independent attempts) rather than pass@k (succeeds on at least one). An agent that solves a task 90% of the time but fails unpredictably is dangerous in an automated loop. Favor consistency across repeated runs, and be suspicious of a headline number reported from the single best sample.
- 5
Build your own weighted rubric with a programmatic anchor
Turn step one into a scoring rubric you control. List the dimensions that matter — task success, tool-call correctness, safety and policy adherence, cost and latency, recovery from errors — and weight them by what your use case actually needs. Anchor the rubric to at least one programmatic, objective check so the whole score is not a matter of opinion; use rubric or judge scoring only for the genuinely subjective parts. A defensible rubric lets you justify a choice to a skeptical stakeholder, which a vendor's single number never will.
- 6
Remember the score is the model plus its harness
The same model can post wildly different benchmark numbers depending on its scaffolding — the prompt, the tool definitions, retries, planning loop, memory and verification steps around it. A benchmark result measures a model-plus-harness combination, not the model alone. When you read a score, ask what harness produced it and whether that resembles the one you will run. A strong result achieved with an elaborate harness you cannot reproduce tells you little about what the model will do inside your own, simpler scaffolding.
- 7
Re-evaluate on your own data before production
Public benchmarks narrow the field; they do not pick the winner for you. Before you commit, run your shortlist on a held-out sample of your own real tasks using your own rubric and harness, and watch for contamination (tasks the model may have seen in training) and distribution shift from the public set. Re-run periodically as models, prompts and your task mix change. The benchmark is where you start; your own evaluation is where you decide.
Browse the major agent benchmarks by task type, see how each one grades, and find the ones that actually match your use case — free, in your browser.
Related
Browse benchmarks by task and grading method to pick the right one for your evaluation.
Eval rubric chain builderBuild the weighted, programmatically-anchored rubric this guide tells you to create.
Agent benchmarks comparedA deeper look at WorkBench, JobBench and BigLaw Bench — what each tests and how each grades.
Harness designerDesign the scaffolding around a model, since the score you get is always the model plus its harness.
Frequently asked questions
Which benchmark should I use to evaluate an agent?
It depends entirely on your task. Use SWE-bench for coding and bug-fixing, tau-bench for tool and API use under a policy, WebArena for web navigation, GAIA or BrowseComp for open-ended research and browsing, and WorkBench for office and knowledge-work automation. The right benchmark is the one whose tasks look like yours; a strong score on a mismatched benchmark tells you almost nothing about how the agent will perform on your work.
Why not just trust the vendor's headline score?
A single number hides too much: which benchmark it came from, whether that benchmark matches your task, how it was graded, whether it is a best-of or an average, and what harness produced it. Vendors also tend to report their most flattering configuration. The number is a starting point for a shortlist, not a decision. You make the decision by evaluating on your own tasks with your own rubric.
What is the difference between pass@k and pass^k?
pass@k measures whether the agent succeeds on at least one of k attempts, so it rewards getting lucky once. pass^k measures whether it succeeds on all k attempts, so it rewards reliability. For anything running in an automated loop, pass^k is the more honest number because an agent that usually works but fails unpredictably can do real damage. A headline score is often a best-of, which flatters an inconsistent agent.
How do grading methods differ in how much I should trust them?
Programmatic grading (tests pass, the expected end-state is reached, the right tool was called) is the most trustworthy because it is deterministic and reproducible. Rubric grading scores against a written checklist and is good for subjective domains but only as reliable as the rubric. Human or LLM-as-judge grading scales to open-ended tasks but adds bias and inconsistency, and an LLM judge can be swayed by verbose or confident answers. Always anchor your own evaluation to at least one programmatic check.
Does the same model always score the same?
No. A benchmark result reflects the model plus its harness — the prompt, tool definitions, retries, planning loop, memory and verification around it. The same model can score very differently with different scaffolding, so two results are only comparable if the harness is comparable. When you read a score, ask what harness produced it and how close that is to the one you will actually run.
Do I still need to test on my own data if the benchmarks look good?
Yes. Public benchmarks narrow the field but cannot account for your specific tasks, data distribution or policies, and some tasks may have leaked into training (contamination), inflating scores. Run your shortlist on a held-out sample of your own real work, graded with your own rubric and harness, before committing — and re-run periodically as models and your task mix change. The benchmark is where you start; your own evaluation is where you decide.
Can an LLM-as-a-judge be manipulated by the content it grades?
Yes, if the judge reads attacker-influenced content. Datadog Security Labs (October 2026) demonstrated a “counterfeit policy” — text prepended to the evaluated content that redefines the grading categories and supplies the desired verdict — collapsing an LLM judge’s prompt-injection recall from about 83% to under 4%, and dropping a tool-risk score from 0.89 (blocked) to 0.29 (allowed). Notably a crude fake “[system]” block did not work; the effective attack rewrote the judge’s own labels and taxonomy. The defences: keep untrusted or user-editable content isolated from the judge’s trusted instructions and category definitions, anchor the score with a programmatic check wherever possible, and never let an LLM judge be the authorization gate — enforce permissions deterministically outside the model.