Ultrafuzz Open-Sourced: Lessons for AI Security Testing

Ultrafuzz — an agentic smart-contract fuzzing and threat-hunting orchestrator — has been open-sourced under the MIT license. Announced by Antonio Viggiano and released under the monad-developers organization, it coordinates hundreds of specialized agents to configure a target repository, model its threats, extract properties, and hunt bugs in parallel. The code is interesting, but the more valuable contribution is a set of four experiments that quietly answer a question everyone building AI-assisted security tooling is asking: what actually makes agentic vulnerability discovery work? The answers generalise well beyond DeFi.

What Ultrafuzz is

At a high level, Ultrafuzz is a graph of specialized agents connected by artifact hand-offs. A run moves through setup and project discovery, pulls in a vulnerability database, defines actors and user flows, prepares a test harness, threat-models the system, specifies properties, fans out into many bug-finding strategies (invariant testing, differential testing, goal-directed hunts), deduplicates and triages the findings, and writes a report. It supports both Solidity and Vyper, both frontier and open-weight models, and ships with zero-configuration “audit profiles” so users do not have to hand-tune every depth and breadth knob.

Crucially, the team built an evaluation harness around it — UltrafuzzBench, plus support for public suites and a few private datasets — so that prompt and topology changes could be measured for precision, recall, F1, time, cost and token usage rather than argued about. That evaluation-driven discipline is what makes their findings worth reading.

The four experiments — and why they matter to anyone building AI security tools

The team ran a series of controlled ablations, each on separate machines so runs did not share caches or state, to test what was really driving the tool’s bug-finding advantage.

1. Forcing every strategy to write fuzz tests was not optimal

The original design instructed agents to turn each property into an executable fuzz test. A variant, “NoFuzz,” kept the same topology but told agents to investigate the property directly (“find a bug that breaks property X”) instead of authoring a harness. Surprisingly, NoFuzz found substantially more bugs — and used fewer tokens doing it — while still catching every bug the fuzz-first version found. The team’s read: capable models can spot many simple, stateless bugs by reading the code, and the overhead of writing a test harness eats context budget that could go toward finding more issues. They adopted direct, property-guided investigation as the default.

2. Stateful fuzzing still earns its place

That did not retire fuzzing. On a different benchmark, executable fuzzing surfaced a bug that the read-the-code approach did not: a conservation invariant that only broke across a sequence — borrow a unit, let time advance, then read the aggregate — producing a one-unit rounding mismatch. Bugs that depend on multiple interactions or a specific ordering need the deeper reasoning and state exploration that stateful fuzzing provides. The lesson is not “fuzz everything” or “fuzz nothing” — it is to reserve executable stateful fuzzing for the bugs that actually benefit from sequences and changing state.

3. Explicit properties raise recall

A property is a concrete, testable claim about how a system should behave — “the sum of all balances equals total supply,” “accumulated fees only ever increase,” “a transfer decreases the sender and increases the recipient by the same amount.” When the team stripped out the property-specification step entirely, recall dropped. Not catastrophically, but measurably. Giving agents specific correctness conditions to investigate — rather than a vague instruction to “find bugs” — is a real force multiplier.

4. Exploration breadth (and compute) is a major lever

Because running the same prompt multiple times yields different results, one of the simplest levers is to spend more compute and run more distinct strategies. Ultrafuzz grew from five strategies to more than twenty, and recall grew with them. This mirrors a broader finding across AI-assisted vulnerability research: throughput and diversity of attempts matter as much as any single clever prompt. It also turns bug-finding into an explicit time-and-money optimisation problem — how do you hold recall steady while minimising spend?

The operational-security lesson everyone should copy

Buried in the announcement is the most important line for defenders: run Ultrafuzz only on ephemeral, isolated virtual machines you can safely discard. The agents run in an unrestricted, skip-permissions workflow, which means they may install or access dangerous tooling or reach sensitive credentials, and prompts, model choices, and target behaviour can steer them toward unintended or destructive actions on the host.

This is the same risk class we have written about with malicious LLM routers and tool-call injection: an agent that can act is a confused deputy waiting to happen. The controls are the same, too — treat the agent host as untrusted and disposable, give it least-privilege and no standing credentials, allow-list its egress, and never point a skip-permissions agent at a developer workstation or a machine holding anything valuable. If you are adopting any agentic security tool, that isolation posture is non-negotiable.

A shared vulnerability taxonomy

Ultrafuzz seeds its hunts from a structured vulnerability database — currently the OWASP Smart Contract Security Project — and the team proposes going further: a graph of bug categories, subcategories and concrete instances that agents can traverse to a chosen depth. A “denial of service” category might branch to “liquidation DoS,” then to specific failure modes like dust positions that block liquidation. The depth becomes a knob for how much compute to spend, and coverage of the taxonomy becomes a measurable target. A common, extensible taxonomy of failure modes is exactly the kind of shared infrastructure that makes automated vulnerability research more systematic — and it is a good reminder that curated threat knowledge, not just raw model capability, drives results.

What security teams should take away

• Combine tools; don’t bet on one. The evals repeatedly showed that different approaches surface different bugs. Layer static review, property-guided AI investigation, and stateful fuzzing rather than assuming any single method is complete.
• Invest in properties and invariants. They help humans and agents alike, and they were one of the clearest levers on recall. If you write smart contracts — or any system with strong correctness conditions — specifying invariants pays off.
• Budget compute deliberately. More diverse strategies and more runs find more bugs. Treat AI-assisted review as a tunable spend, not a one-shot.
• Sandbox agentic tooling by default. Skip-permissions agents belong on throwaway, credential-free VMs with constrained egress — full stop.
• Red-team your own models and agents. Before you trust an agent to touch your code or infrastructure, probe how it behaves under adversarial input. Our free Model Audit runs jailbreak, injection and secret-leak checks against an endpoint you control, and MCP Guard audits agent/MCP configs for the misconfigurations that make an agent dangerous.

The broader signal is one we keep returning to in our writing on how AI misuse is detected and disrupted: agentic systems are moving fast, they are powerful, and the discipline that makes them safe and effective — evaluation, isolation, least privilege, and shared threat knowledge — is exactly the discipline security teams already know. Ultrafuzz is a useful, open contribution precisely because it shows that work in the open.

Frequently asked questions

Common questions about Ultrafuzz and what its experiments mean for AI-assisted security testing.

What is Ultrafuzz? An open-source, agentic smart-contract fuzzing and threat-hunting orchestrator announced by Antonio Viggiano and released under the monad-developers organization on GitHub. It coordinates hundreds of specialized AI agents to configure a repository, define actors and flows, extract properties, threat-model the system, run parallel bug-finding strategies, build stateful and stateless fuzz tests, deduplicate findings and produce a report. It supports Solidity and Vyper and both frontier and open-weight models, under the MIT license.

What did its experiments actually find? Four things: forcing every strategy to write fuzz tests was less efficient than direct, property-guided investigation for their target; stateful fuzzing still earns its place for sequence- and time-dependent bugs; explicitly specifying properties and invariants improves recall; and exploration breadth — more diverse strategies and more compute — is one of the strongest levers on bug count.

Does this mean fuzzing is dead and AI can just read the code? No. The team was explicit that fuzzing is not generally ineffective — only that requiring every strategy to author a harness was suboptimal for that target. Genuinely stateful bugs still needed executable fuzzing, so the final design uses property-guided investigation broadly and reserves stateful fuzzing for bugs that benefit from sequences and changing state.

Is it safe to run an agentic security tool like this? Only with strict isolation. Ultrafuzz’s own guidance is to run it only on ephemeral, isolated virtual machines you can discard, because the agents run in an unrestricted, skip-permissions workflow and may install or access dangerous tooling or sensitive credentials. Never run it on a developer workstation, a persistent environment, or any machine holding valuable data.

What are the takeaways for security teams outside DeFi? Different tools find different bugs, so combine approaches; invest in clear properties and invariants; budget compute deliberately because more runs find more bugs; sandbox any agentic tool that runs with skipped permissions; and use a structured taxonomy of vulnerability classes to give agents a systematic starting point.