Eval Rubric-Chain Builder
A benchmark or eval score is only as trustworthy as its grader. Build the grader here: compose an ordered chain of criteria — each weighted, each graded by a programmatic check, a written rubric, or a human/LLM judge, some of them hard gates — then read back how much of the score rests on reproducible checks versus subjective judgement, preview the weighted scorecard on a candidate, and catch the integrity traps before they ship. Start from a template that mirrors how real agent benchmarks grade. Everything runs in your browser; nothing is uploaded, and no benchmark numbers are invented.
State-match + policy gate style (as in tool-and-database agent benchmarks): the end state is checked in code, policy is a hard gate, tone is the one subjective line.
Rubric
The chain
Criteria run top to bottom. A hard gate that fails breaks the chain and zeroes the composite score — use it for safety and no-fabrication checks. Weights are normalized to 100% for scoring.
Programmatic check: A deterministic check in code — unit/integration tests pass, the final state matches, an exact-match or regex/JSON-schema assertion holds. Same input, same verdict, every time; no judge in the loop.
Programmatic check: A deterministic check in code — unit/integration tests pass, the final state matches, an exact-match or regex/JSON-schema assertion holds. Same input, same verdict, every time; no judge in the loop.
Programmatic check: A deterministic check in code — unit/integration tests pass, the final state matches, an exact-match or regex/JSON-schema assertion holds. Same input, same verdict, every time; no judge in the loop.
Programmatic check: A deterministic check in code — unit/integration tests pass, the final state matches, an exact-match or regex/JSON-schema assertion holds. Same input, same verdict, every time; no judge in the loop.
Human / LLM judge: A human or an LLM-as-judge forms an opinion with no fixed rubric. Right for taste, tone and nuance, but subjective: it drifts between graders and model versions, so a score built on it is directional, not reproducible.
Undisclosed rubric — graders cannot apply it consistently and readers cannot audit it.
Scorecard preview
Enter a candidate's result per criterion to see the weighted total this rubric would produce. For a pass/fail criterion, toggle pass or fail; for a scored one, enter 0–100. A failed gate zeroes the whole score.
Grader trust
heuristicMost of the score (90% of weight) rests on reproducible programmatic checks. This grader is hard to game and easy to re-run.
An illustrative reproducibility heuristic from the grading methods and weights you chose — not a measured inter-rater figure and not a benchmark result.
Warnings
1“Tone & empathy” use a rubric or human/LLM judge whose grading standard is not disclosed. Graders cannot apply it consistently and readers cannot audit the score — write the rubric down and publish it.
Export
The export bundles the chain, the grader-trust read and every warning. Everything runs in your browser; nothing is uploaded.
What this tool is
- A free, fully in-browser builder for a defensible evaluation rubric for an agent: compose an ordered chain of criteria, each with a weight, a grading method and a pass/fail or scored mode, and see what the score would actually mean. Nothing is uploaded; there are no server calls.
- Each criterion is graded one of three ways — a programmatic check (code, test-suite, state-match, exact-match), a written rubric, or a human/LLM judge. A hard gate (safety, no-fabrication) breaks the chain and zeroes the score when it fails.
- Start from a template that mirrors real agent-benchmark grading styles — customer-service, coding, research, or legal/professional work — then edit every criterion, weight and gate.
- The “grader trust” read is the point: a reproducibility heuristic showing how much of the score rests on reproducible programmatic checks versus subjective human/LLM judgement — because a score is only as good as its grader.
- Warnings flag the integrity traps: undisclosed-rubric criteria, 100% LLM-judge grading, no programmatic anchor, weights that do not sum to 100 (normalize in one click), and a scored criterion used as a hard gate. Export the rubric as Markdown or JSON, or share a permalink.
A score is only as good as its grader
It is easy to report a single evaluation number and much harder to defend what it means. The same agent transcript can score 95 or 40 depending on which criteria you grade, how you weight them, and — most of all — how each criterion is decided. A programmatic check that runs a test suite or compares the final state gives the same verdict every time and cannot be talked out of it. A written rubric is reproducible only to the degree it is specific, disclosed and graded consistently. A human or LLM judge with no fixed rubric is the right tool for taste and nuance, but its verdict drifts between graders and model versions. A rubric that is 100% judge is not a benchmark; it is an opinion poll of one.
This builder forces those choices into the open. The grader-trust read splits your rubric's weight across the three methods and gives a reproducibility heuristic, so you can see at a glance whether the score has a programmatic anchor or rests entirely on judgement. Hard gates let you encode the failures that must count — a safety violation, a fabricated citation, a patch that does not pass its tests — so a strong average cannot mask an unacceptable outcome. And the warnings name the traps that quietly undermine an eval: an undisclosed rubric no one can reproduce, judgement-heavy grading reported without inter-rater agreement, weights that do not sum to 100. The theme runs straight through the Agent Benchmark Explorer and the Harness² line of work (Google Cloud AI Research and Northwestern University, arXiv 2610.14224): the scaffolding around the model, grading included, is where reliability is won or lost.
Frequently asked questions
What is an eval rubric chain, and why build one?
An evaluation rubric is the set of criteria you grade an agent against; a chain is those criteria in a deliberate order, some of them hard gates. Building one explicitly matters because a benchmark or eval score is only as trustworthy as its grader: the same agent output can score very differently depending on which criteria you grade, how you weight them, and whether each criterion is checked by code, by a written rubric, or by a human or LLM forming an opinion. This tool makes those choices visible so the resulting number is defensible rather than a black box.
What is the “grader trust” read?
It is a reproducibility heuristic. Each grading method reproduces differently: a programmatic check (a test suite, a final-state match, an exact-match assertion) gives the same verdict every time with no judge in the loop; a written, disclosed rubric reproduces only as well as it is specific and graded consistently; an ungoverned human or LLM judge is subjective and drifts between graders and model versions. The tool weights each criterion by its method and reports how much of the total score rests on reproducible programmatic checks versus judgement. It is an illustrative design heuristic computed from your choices — not a measured inter-rater figure and not a benchmark result.
What does a hard gate do?
A hard gate is a criterion that breaks the chain when it fails: if it fails, the composite score is zero no matter how well everything else did. Gates are for the things that must hold regardless of quality — a safety or policy violation, a fabricated citation or legal authority, a patch that does not pass its target tests. Modeling them as gates rather than just heavily-weighted criteria stops a strong score on everything else from masking an unacceptable failure. The tool flags a gate that uses a scored (rather than pass/fail) criterion, because a gate reads cleaner as an unambiguous cut-off.
Does this invent benchmark numbers or cite specific benchmark scores?
No. The templates mirror the grading STYLES catalogued in the Agent Benchmark Explorer — state-match and test-suite checks, exact-match answers, rubric grading of open-ended work — but they contain no reported scores, task counts or benchmark figures. The only numbers the tool produces are the weighted scorecard preview from results you enter yourself, and the grader-trust reproducibility heuristic, both of which are computed live from your own inputs and labelled as heuristics.
Does anything leave my browser?
No. The whole tool is client-side React with inline SVG and static data. There are no network or API calls, no account, and nothing you configure is sent anywhere. The shareable link encodes your rubric in the URL hash (base64url), so it round-trips entirely in the browser, and the Markdown and JSON exports are generated locally.