Agent Harness Comparator
Before you compose a harness, you have to choose an approach. This puts the credible agent-harness architectures side by side — Harness² (Google Cloud AI Research, arXiv:2610.14224), ReAct, Reflexion, DSPy, GEPA, ADAS and a single-prompt baseline — across the dimensions that decide adoption: self-improvement mechanism, recursion and meta-optimization, verifier, routing, tool & egress model, human-in-loop, data needed, a relative (not dollar) cost multiplier, security surface and maturity. Filter the columns, highlight how each differs from a reference, or ask it for the best fit for your job. Everything runs in your browser; cells say “—” where a method simply does not define a property, and the only scored parts are labeled heuristics, not benchmarks.
Best fit for…
heuristicPick the job to be done and get an editorial recommendation. This is a judgement call, not a measurement — it sets the reference column and highlights how the rest differ.
| Dimension | Harness²REF Google Cloud AI Research · arXiv:2610.14224 | ReAct Yao et al., 2022 (ICLR 2023) · Princeton + Google | Reflexion Shinn et al., NeurIPS 2023 | DSPy Stanford NLP (Khattab et al.), 2023–2024 | GEPA Agrawal et al., 2025 · a DSPy optimizer | ADAS Hu, Lu & Clune, 2024 | Single-prompt baseline No harness — the reference point |
|---|---|---|---|---|---|---|---|
Self-improvement mechanism How (and whether) the system gets better over time, and what exactly changes. | The harness itself is optimized. A winning configuration and its evidence-backed lessons persist into an "improvement harness" that configures the next task — learning across tasks without touching model weights. | None across runs. Within a run it grounds reasoning in tool observations, but nothing is learned for next time. | Verbal self-reflection: on failure it writes a natural-language lesson to episodic memory and retries. Improves within an episode; no weight update. | Compile-time optimization: optimizers (bootstrap few-shot, MIPRO, …) tune prompts/weights of a fixed program against a metric. Improvement happens offline, not during a run. | Reflective prompt evolution: mutate prompts, reflect in natural language on execution traces, and Pareto-select survivors across rounds. Offline, prompt-level improvement. | A meta-agent writes new agents in code and searches a growing archive of designs (Meta Agent Search). The unit being improved is the agent design itself. | None. |
Recursion / meta-optimization Does it iterate within a task, and does anything optimize the scaffolding itself? | Two nested loops: a task-level Propose·Probe·Compose fan-out, and a harnessing-level meta-loop that rewrites the scaffolding across tasks. Explicit meta-optimization. | Iterates thought→action→observation within a single task. No meta-optimization of the scaffolding. | Sequential retry loop driven by reflection. No meta-optimization of the harness itself. | Optimizes the pipeline offline; the runtime graph is fixed and does not self-modify. Meta-optimization is bounded to compile time. | Evolutionary/sequential optimization rounds over prompts. Offline optimizer (runs as a DSPy optimizer); the runtime program stays fixed. | Explicit meta-level search over agent architectures — the closest public analogue to the Harness² outer loop. | None — a single forward pass. |
Verifier / eval loop What sits between a draft and an action or an accepted answer. | Yes — the Probe step evaluates candidate approaches before the Compose step selects/merges them. | No dedicated verifier — the environment / tool observations are the only check. | Yes — a success/failure signal (environment reward, test, or heuristic evaluator) decides when to reflect and retry. | Metric-driven, and assertions/suggestions can gate outputs at runtime. The optimizer scores candidates on a dev set. | A metric plus textual feedback drawn from execution traces guides which candidates survive. | Candidate agents are scored on a target benchmark; that score drives the search. | None. |
Model routing Whether cheap-draft / strong-verify routing is part of the method. | — (not a defined part of the published method) | — | — | Can target multiple LMs per module, but routing is not the headline mechanism. | — | — | Single model. |
Tool / egress model How broad the tool surface is and whether network/filesystem reach is constrained. | Broad tool surface (tools, subagents, skills, scripts, plugins). Egress containment is not a published control. | Tool-using by design; typically a scoped toolset. Egress depends entirely on which tools you expose. | Inherits a ReAct-style scoped toolset; egress depends on the exposed tools. | Scoped — typed signatures and explicit modules; the surface is whatever tools the program declares. | Scoped — inherits the underlying program’s declared tools. | Generated agents are defined and run as code, so the effective surface is broad (code execution of discovered designs). | No tools, no egress — just the model and the prompt. |
Human in the loop Where a person is required by the method (vs. optional wiring you add). | None required — the improvement loop is autonomous by design. | None required (you can insert approval before actions). | None required by the method. | The author defines the metric and signatures; not in the runtime loop. | The author supplies the metric/feedback; not in the runtime loop. | None in the search loop by default. | Whatever the user does at the prompt; nothing programmatic. |
Data needed to optimize What you must supply for the method to improve anything. | A distribution of tasks with a verifiable success signal to drive the improvement loop. No labeled weight-training set. | None to run — a few in-context exemplars are enough. Nothing to "optimize". | A per-attempt success/failure signal (a reward function or test). No training set of examples. | A training/dev set of examples plus a metric to compile against. More data generally gives a better compile. | A task set and a metric, ideally with textual feedback on traces. Reported to be more sample-efficient than RL-style tuning. | A benchmark / evaluation set to score each discovered design during the search. | None. |
Typical cost multiplier Relative, qualitative inference cost vs. the single-prompt baseline. HEURISTIC — no $ figures. | High — multiple candidate branches per task, plus the outer optimization loop. | Low — one interleaved loop, a handful of extra tool/observation turns. | Moderate — several sequential attempts, each with a reflection pass. | Compile-time cost (the optimization run); low and predictable at runtime. | Optimization-time cost, aimed at fewer rollouts than RL; low at runtime. | High — searching over many generated-and-evaluated agents. | 1× — the reference point. |
Security surface The main attack surface and governance concern the architecture introduces. | Highest. A harness that can rewrite its own prompt, tools and guardrails can quietly expand its own authority; broad tools and open egress by default enlarge the blast radius. | Indirect prompt injection via tool outputs/observations, and whatever the exposed tools can reach. Small surface if tools are tightly scoped. | Memory poisoning: unprovenanced reflections can carry a bad "lesson" forward across attempts; plus the underlying tool surface. | Small at runtime — a fixed, typed graph. Concerns move to the optimizer metric (gameable) and the module/library supply chain. | Evolved prompts should be reviewed before promotion; otherwise the runtime surface is the fixed underlying program. | High. A meta-agent that writes and runs code produces arbitrary, unreviewed agent designs — code execution and self-expansion are inherent. | Smallest — prompt injection via the single prompt only. No tools, memory or egress to abuse. |
Maturity / origin Where it comes from and how settled it is. | Research — Google Cloud AI Research, arXiv:2610.14224. | Established and widely adopted — Yao et al., 2022 (ICLR 2023). | Established research — Shinn et al., NeurIPS 2023. | Established open-source framework — Stanford NLP, 2023–2024. | Recent research — Agrawal et al., 2025; implemented as a DSPy optimizer. | Recent research — Hu, Lu & Clune, 2024. | Universal baseline. |
Cells marked “—” are properties the method does not define, not gaps we guessed at. The coloured left-borders on the cost and security rows are HEURISTIC shading (green = lighter, red = heavier), not benchmark results, and no exact costs are claimed. Harness² is attributed to Google Cloud AI Research, arXiv:2610.14224. Verify against each method’s primary source before adopting.
Security surface, by approach
The architecture you adopt decides the attack surface you inherit. Each callout links to the PlayCISO lab that drills into that surface.
Security surface: Highest. A harness that can rewrite its own prompt, tools and guardrails can quietly expand its own authority; broad tools and open egress by default enlarge the blast radius.
Security surface: Indirect prompt injection via tool outputs/observations, and whatever the exposed tools can reach. Small surface if tools are tightly scoped.
Security surface: Memory poisoning: unprovenanced reflections can carry a bad "lesson" forward across attempts; plus the underlying tool surface.
Security surface: Small at runtime — a fixed, typed graph. Concerns move to the optimizer metric (gameable) and the module/library supply chain.
Security surface: Evolved prompts should be reviewed before promotion; otherwise the runtime surface is the fixed underlying program.
Security surface: High. A meta-agent that writes and runs code produces arbitrary, unreviewed agent designs — code execution and self-expansion are inherent.
Security surface: Smallest — prompt injection via the single prompt only. No tools, memory or egress to abuse.
What this tool is
- A free, fully in-browser comparison matrix for agent-harness architectures: Harness², ReAct, Reflexion, DSPy, GEPA, ADAS and the single-prompt baseline, side by side. Nothing is uploaded; there are no server calls.
- Rows are the dimensions that actually decide adoption — self-improvement mechanism, recursion / meta-optimization, verifier / eval loop, routing, tool & egress model, human-in-loop, data needed to optimize, a relative (not dollar) cost multiplier, security surface, and maturity / origin.
- Every factual cell is written from the public description of each method. A cell reads “—” where the property is simply not part of that method — never a guess — and the only scored elements (the best-fit recommendation and the cost / security shading) are labeled HEURISTIC.
- Pick which approaches and dimensions to show, choose a reference column and highlight how the others differ, or use the “best fit for…” selector to get a one-line recommendation for a given job.
- Share the exact view as a base64url permalink, and export the whole comparison as Markdown or as a standalone HTML file — both generated locally in your browser.
Choosing an approach is choosing a security posture
The headline question — “which agent architecture should we adopt?” — is usually argued on capability. But the dimensions that bite in production are the ones this matrix makes explicit next to each other. A method that optimizes offline against a metric and then freezes a typed graph (DSPy, GEPA) leaves a small, reviewable runtime surface. A method that rewrites its own scaffolding across tasks (Harness²) or writes and runs new agents in code (ADAS) is more powerful and carries a far larger attack surface and a self-modification problem you have to gate. The same lens applies to cost: parallel branches and outer optimization loops multiply inference, so the cheapest thing that clears the bar is often the right call.
That is why security surface sits in the matrix as a first-class row and gets a per-approach callout, each linking to the PlayCISO lab that drills into it — tool scope, memory governance, egress containment, recursive change-control, evaluation integrity. The comparison is deliberately honest about uncertainty: a cell reads “—” when the property is not part of the method rather than filling the gap with a plausible guess, and anything we score — the best-fit recommendation, the relative cost multiplier, the green/amber/red shading — is called a heuristic, because it is one. Harness² is attributed to Google Research (arXiv:2610.14224); the other methods to their own authors, listed in each column.
Frequently asked questions
How is this different from the Harness Designer?
The Harness Designer is a composer: you assemble one harness from the eight Harness² components, tune the control loop, and get a diagram, a thesis and a governance-weighted scorecard for that single design. The Harness Comparator is a comparison matrix: it puts whole named architectures — Harness², ReAct, Reflexion, DSPy, GEPA, ADAS and a single-prompt baseline — side by side across concrete dimensions so you can decide which approach to adopt in the first place. Use the Comparator to choose a family, then the Designer to compose and score a specific instance of it.
Are the comparisons benchmark results?
No. Every factual cell is written from the public description of each method — how it improves, whether it recursively optimizes its own scaffolding, what verifier it uses, what data it needs, and so on. The tool quotes no benchmark numbers and invents no exact costs: the cost row is a relative, qualitative multiplier against the single-prompt baseline, and both it and the security-surface shading are labeled HEURISTIC. Where a method simply does not define a property (for example, model routing is not part of ReAct), the cell reads “—” rather than a guess.
What is Harness² and who is it from?
Harness² is Google Cloud AI Research’s self-improving agent harness, described in arXiv:2610.14224. Its central idea is that the scaffolding around a model — the system prompt, guardrails, memory, tool guidance, subagents, skills, scripts and plugins — is what you tune to improve an agent, and that this scaffolding can be optimized recursively: a winning harness and its evidence-backed lessons persist into an improvement harness that configures the next task. It is the most capable and the most governance-hungry architecture in the matrix, which is why security surface is treated as a first-class row here.
Why is security surface a column-by-column concern?
Because the architecture you adopt decides the attack surface you inherit. A single-prompt baseline can only be prompt-injected; a ReAct agent adds indirect injection through tool observations; Reflexion adds a poisonable reflection memory; a meta-agent like ADAS writes and runs generated code; and a self-improving harness like Harness² can rewrite its own prompt, tools and guardrails. The tool gives each approach a security-surface cell and a callout that links to the PlayCISO lab that drills into that specific surface, so the comparison is a security decision as much as a capability one.
Does anything leave my browser?
No. The whole tool is client-side React with a static dataset. There are no network or API calls, no account, and nothing you select is sent anywhere. The shareable link encodes your selection in the URL hash, and the Markdown and HTML exports are generated and downloaded entirely in the browser.