Agentic Pentesting Systems Compared: 2026 Master Guide

Agentic pentesting is penetration testing in which an AI agent, not a person, plans the test, picks and runs the tools, reads the output and decides what to do next. This guide maps the systems that do it: open-source agents, operator-led harnesses, tool-wrapping servers, commercial platforms and validation platforms, plus the AI code-vulnerability hunters next door. Each row in the comparison table links to the project's or vendor's own page, checked on 2 November 2026.

Who this is for: CISOs, heads of security testing, AppSec and red-team leads choosing, buying or governing these tools.

PlayCISO is not affiliated with any system listed. Descriptions come from each project's own README or the vendor's own site. We did not run these tools. No commands, payloads or step-by-step exploitation appear here.

What "agentic pentesting" means, and what it is not

The practical difference from a scanner is the one Shannon's README states plainly: it executes real exploits that can change application state, and it says "Do not run Shannon against production systems". Most products in this group, whether open-source or commercial, try to prove a finding. That cuts false positives and raises operational risk at the same time.

Five architecture types, and how to choose

1. Autonomous open-source agents. You run them yourself with your own model key. Strix describes a "Graph of Agents" with specialised recon, exploitation and post-exploitation agents. Shannon runs a pipeline from reconnaissance and source analysis to exploitation, validation and reporting inside an ephemeral Docker container. PentAGI splits work across a team of agents that run Kali tools in sandboxes. PentestGPT 1.0 drives Claude Code or Codex through a multi-stage pipeline. Choose when you have an engineer to operate it, a lab or staging target, and a way to control model spend.
2. Operator-led harnesses. A human runs the engagement and agents work inside it. Nebula organises an engagement into a Project and lets the operator set scope, approvals and budgets. Executioner is a desktop app that drives agent command-line tools you already have (Claude Code, Codex, Gemini and others) and says "You decide what runs". Choose when your testers are people you trust and you want AI to remove toil, not replace judgment.
3. Tool-wrapping MCP servers. HexStrike AI exposes a large set of security tools to any MCP-compatible AI assistant. The intelligence is whichever assistant you connect, and the tools must be installed separately. Choose when you want to experiment with an assistant that can operate a toolbox, in a lab. Note that Google's GTIG lists Hexstrike among frameworks abused in incident-response engagements.
4. Commercial continuous-testing platforms (SaaS). XBOW, Escape, Aikido Attack, RunSybil, Terra Security and Ethiack sell testing as a service, mostly for web apps and APIs, with scope, scheduling and reporting built in. Choose when you need continuous coverage and audit-ready reports without operating the agent yourself, and you can accept a vendor holding your target data.
5. Validation platforms. Horizon3 NodeZero and Pentera test networks, identity and cloud by safely exploiting what they find, then verify fixes. Choose when the question is "which weaknesses can an attacker actually chain in my environment", not "is this app secure".

A fast way to choose: source code and a staging copy of one web app, with an engineer on hand: open-source agents. Many apps and a small team: a commercial platform. Internal network and identity risk: a validation platform. A skilled red team that wants speed: an operator-led harness. A library with fuzz harnesses, or a codebase to review: a code-vulnerability hunter.

Master comparison table

One row per system. "Claim" means published by the vendor or maintainer. "Checked" is 2 November 2026 for every row. Star counts are a snapshot from that day's page.

Flagged: frameworks seen abused or with a dual-use track record

• ARTEX (open-source, multi-agent, Go): GTIG says threat actors used it in real intrusions and that its creator took the original repository down in October 2026, though forks continue. See What GTIG found. We do not recommend it as a testing tool.
• Villager: published on PyPI by Cyberspike-linked accounts, an MCP client integrating Kali tools and DeepSeek models. Straiker, as reported by The Hacker News on 15 September 2025, found remote-access-tool components and warned of a Cobalt Strike-style trajectory; the package had nearly 11,000 downloads. Source.
• CAIRN: named by GTIG as abused; see Operation CAIRN. Also in our archive: VIPER and Claude-Red.

What the published results actually show

One third-party test, with caveats. Doyensec compared Aikido's Attack AI Pentest and XBOW's Lightspeed on two open-source web apps and validated every finding by hand, using a different researcher per platform. Aikido reported 49 true positives and 2 false positives; XBOW 31 and 1. Severity was assigned correctly 69% vs 68% of the time. Both bought the $4,000 tier. Doyensec concluded the differences were "generally incremental rather than stark", noted Aikido's faster and easier setup (under 20 minutes; XBOW took several days and over 22 support emails for one app), and gave XBOW "marginal benefits in precision". Its limits: Aikido sponsored the study, two apps, one configuration, no false-negative measure, and it said readers should treat it "as a test drive". Study.

A vendor reply, not a replication. Keygraph ran Shannon v3 on one of those apps (Photoview 2.4.0) with three models and compared against Doyensec's figures. Keygraph reports 18 true positives for $6.10 (DeepSeek v4 Flash), 10 for $35.07 (Grok 4.6) and 23 true plus 1 false for $115 (Claude Opus 5). Keygraph verified its own findings, built its patch ground-truth by keyword search of commit history, and says its timing comparison is not like-for-like. Keygraph document.

XBOW's bug-bounty record. XBOW's own post says it reached the top of HackerOne's US leaderboard and submitted "nearly 1,060 vulnerabilities". The outcome categories it lists (resolved, triaged, duplicate, informative and others) sum to 1,044, not 1,060, and 208 were duplicates and 209 informative. It also says its security team reviewed findings before submission to comply with HackerOne's automated-tool policy. XBOW.

Benchmark scores. PentestGPT's 86.5% is on an XBOW validation suite and its own README calls it a historical result. Challenge suites test lab targets. They do not tell you how an agent behaves on your estate.

What agents are good at, and where they are weak

Only statements with a source:

• Finding new bugs in modern web apps of low-to-medium complexity. Doyensec says both platforms did this.
• Keeping false positives low by proving exploits. Shannon reports only findings with a working proof of concept; XBOW says validators confirm findings (its post describes a headless browser confirming cross-site scripting); the Doyensec false-positive ratios were 4% and 3% of true positives.
• Speed and persistence. GTIG saw ARTEX run in bursts of roughly 2 to 25 seconds and retry after errors. It also says these frameworks "trade stealth for velocity" and are noisy.
• Cheap repeat testing. Keygraph's cost comparison is the vendor's own arithmetic; treat it as indicative.

Weak or unproven:

• Coverage. Doyensec did not measure false negatives and says more apps could change results.
• Safety on live systems. Shannon warns against production; CAI's authors say the archived code has weaknesses they will not fix, including prompt-injection classes.
• Reliability of setup. Doyensec reported that XBOW's scan paused or crashed multiple times in its tests. That is one study's experience of one product.
• Headline metrics without method. HexStrike's README gives detection and false-positive rates with no stated method.
• Human comparison. No source we opened compares these systems to skilled human testers on equal terms.

Governance and risk for security leaders

• Authorisation. Written permission from the asset owner, per target, before any agent runs. Every project's README restricts use to systems you are authorised to test. An IP filter inside a tool is not authorisation. Use the penetration-test scoping checklist (Pro, free to try).
• Scope enforcement outside the model. Products differ: XBOW lets you set scope, rate limits and windows; Pentera lists throttling, impact limits and an emergency stop; Nebula lets the operator set scope, approvals and budgets. For open-source agents, enforce scope at the network layer, not by prompt.
• Human gates on irreversible actions. Strix and Shannon describe no per-action approval in the pages we read; Executioner and Nebula are built around operator control. Decide where a human must click yes, using the Autonomy vs Oversight Lab (Pro, free to try).
• Data handling and model-provider exposure. With a bring-your-own-key tool, target data, source code and findings travel to your model provider. Strix says source code is never stored or used for training; Pentera says data is not used for training. Those are vendor statements, so ask for contract terms. PentestGPT sends anonymous usage telemetry by default unless you opt out; its README says command outputs, credentials and flag values are not sent.
• Prompt injection. An agent that reads attacker-controlled content can be steered by it. CAI's authors document this class of weakness in their own framework. Isolate the runtime and limit its credentials.
• Logging. Record every command and request the agent issues. XBOW and Pentera advertise audit logs; with open-source tools you must build this yourself.
• Dual-use and supply chain. The tools you evaluate can also be the tools used against you (GTIG; Straiker). Pull from official repositories, check licences (AGPL and research-only terms differ), and watch for tampered clones.

What security leaders should do this week

• Decide which of the five types fits your next test before you talk to vendors.
• Ask every vendor for method, not scores: which targets, which model, who validated, what the false-negative rate is.
• Pilot in a lab first with a deliberately vulnerable app you own, with egress limited to the target.
• Write the rules of engagement for agents: scope, hours, rate, destructive-action approval, evidence retention, kill switch.
• Brief your SOC that authorised agent traffic will look like an attack; agree a way to tell it apart. The Human or Agent Lab trains this.
• Measure your exposure to the hostile version with the Agentic Attack Readiness Scorecard.

Series: individual analyses and comparisons

This is the hub. Deeper posts are linked below as they exist.

Published

• PentAGI explained: what it does and is it safe to run?
• Autonomous AI pentest frameworks: how they work and what to ask
• ARTEX agentic hacking framework: what GTIG found
• Operation CAIRN: an LLM-driven attack platform found in the wild
• VulnHunter explained
• AegisLoop (ARTEX-KO Plus): turning a vulnerability into a detection
• VIPER red team platform and how defenders should respond
• Claude-Red: governing an offensive AI toolkit
• Executioner: a local agentic pentesting environment, and what to weigh
• How to use Executioner safely
• Pentest tools compared (data table, 25 tools)

Planned (not yet published)

• Strix explained, and Shannon explained
• Strix vs Shannon, and open-source agents compared
• XBOW vs Aikido: what Doyensec's study does and does not show
• A buyer's guide to commercial agentic pentesting
• A defender's guide to detecting agentic pentest traffic

Frequently asked questions

What is agentic pentesting?

Agentic pentesting means a large language model (LLM) agent, or a team of agents, plans a security test, chooses and runs tools, reads the results and decides what to do next toward a stated goal, with little step-by-step human steering. Unlike a scanner, which runs fixed checks and reports, an agentic system can chain steps and, in most products, try to prove a finding with a working exploit. Whether a human approves each step varies by product.

What is the difference between agentic pentesting and a vulnerability scanner?

A scanner runs predefined checks or templates and reports what matches. An agentic pentester decides its own next action, adapts to what it finds and usually tries to validate a finding by exploiting it. That makes it more capable of finding business-logic flaws, and also a different risk class: it can change application state, so Shannon's own README says never to run it against production.

What are the best open-source agentic pentesting tools?

The most visible open-source projects we checked are Strix (Apache-2.0), Shannon (AGPL-3.0), PentAGI (MIT), PentestGPT (MIT), HexStrike AI (MIT), Nebula (BSD-2-Clause) and Executioner (MIT). They differ in architecture and in what they target, so "best" depends on your use: web apps with source code, infrastructure, CTF practice, or an operator-led desktop workflow. CAI was archived by its authors on 28 August 2026.

Are agentic pentest tools free?

The open-source projects are free to download, but each needs an LLM to run: you supply an API key for a hosted model or point it at a local model, so model usage is a cost you pay. Several also sell paid or hosted tiers (Strix has a cloud at app.strix.ai; Shannon has paid Pro and Enterprise tiers). Commercial platforms such as XBOW, Pentera and Horizon3 NodeZero are paid products; Aikido markets a free start.

Is XBOW better than the open-source tools?

There is no independent head-to-head test that includes both. The only third-party study we found, by Doyensec in May 2026 and sponsored by Aikido, compared Aikido's Attack AI Pentest with XBOW's Lightspeed on two open-source apps. Keygraph, the maker of Shannon, published its own comparison of Shannon against Doyensec's numbers, which is a vendor run and not independent.

Can an AI agent replace a human penetration tester?

The evidence does not show that. Doyensec's study explicitly did not compare AI with human testers, did not measure false negatives, and covered two low-to-medium complexity web apps. Treat agentic tools as a way to test more often and more cheaply, with a human accountable for scope, review and sign-off.

How do I run an agentic pentester safely?

Get written authorisation and a defined scope first. Run it in an isolated environment against non-production systems where the tool says so, enforce scope outside the model, keep a human approval gate on destructive actions, log every action the agent takes, and decide which model provider may see your code and findings. Our governance section below expands each point.

Are attackers using agentic pentesting tools?

Google Threat Intelligence Group and Mandiant reported on 30 October 2026 that threat actors have operated open-source frameworks including ARTEX, PentAGI and CAIRN in real intrusions since late August 2026. Separately, Straiker reported that the Villager framework, distributed on PyPI, bundles components of a remote access tool. Dual-use is the central risk of the whole category.

How do I detect agentic pentest traffic in my environment?

Look at behaviour rather than static indicators: bursts of management API calls and permission tests faster than a human could issue them, new authentication origins, and first contact from servers with webhook or out-of-band services. Mandiant's guidance and our ARTEX analysis cover this; a dedicated defender's guide is planned in this series.

Sources

Primary sources first; all opened on 2 November 2026 unless noted. Where a page blocked our fetch, we say so in the text.

• Google Threat Intelligence Group and Mandiant: ARTEX and Friends (30 October 2026)
• Doyensec: Comparing AI Application Security Testing Platforms, Aikido vs XBOW (27 May 2026) and its blog post
• Shannon (Keygraph) repository and Keygraph benchmark document
• Strix repository and strix.ai
• PentAGI repository
• PentestGPT repository
• CAI (Alias Robotics) repository, archived 28 August 2026
• HexStrike AI repository
• Nebula repository
• Executioner repository
• XBOW and XBOW: how it reached the top of HackerOne (24 June 2025)
• Horizon3.ai NodeZero; Pentera; Escape; Aikido Attack; RunSybil; Terra Security; Ethiack
• Buttercup repository and Trail of Bits announcement (8 August 2025)
• OpenAI: Codex Security research preview and Aardvark announcement (OpenAI pages returned an access error to our fetch; details come from search-result summaries of those pages)
• Google: cybersecurity updates, summer 2025 (Big Sleep)
• The Hacker News: Villager (15 September 2025)
• VulnHunter repository