T3MP3ST: Open-Source AI Pentest Agent Claims 90.1%

T3MP3ST is an open-source framework that turns a coding agent or a local model into an automated security tester, and its README claims a 90.1% score on XBOW's 104-challenge web benchmark, against 85% that XBOW reported for itself. We read the README on 5 November 2026 and confirmed what the tool is and what it says. The comparison is the project's own claim: we found no independent reproduction, and the two scores come from different setups.

Who this is for: CISOs, heads of security testing, AppSec and red-team leads, and anyone who lets coding agents run on company machines.

We opened the T3MP3ST README, XBOW's two blog posts on its benchmark and the benchmark repository's README on 5 November 2026. We did not run any of these tools and did not open the project's linked methodology files. We could not open the social-media post that prompted this story, so its wording is not used as evidence. The repository page shows no release date, so we date this post by the day we checked. PlayCISO is not affiliated with any project or person named. This post gives no commands, setup steps or payloads.

What happened

The GitHub account elder-plinius, associated with the researcher known as Pliny for AI jailbreak work, published T3MP3ST as an open-source red-teaming platform. Its README presents a recon, exploit and report chain that can run either on a coding agent already installed on the machine or on a local model, and it leads with benchmark results. At the time we looked the repository showed 204 commits on its main branch, no published releases and a few thousand stars, and several forks exist. We could not confirm who runs it or how it is used.

Timeline

Why 90.1% against 85% is a weak comparison

• Both numbers are self-reported. The project says XBOW self-reports 85%, and XBOW's own post is the origin of that figure. We found no outside party that re-ran either.
• Different conditions. A score depends on the agent design, the model, how many attempts are allowed, time limits, what the agent is told and how success is graded. XBOW's 2024 post gives none of the model or attempt details, so a like-for-like reading is not possible from public pages.
• An old, saturated benchmark. XBOW now calls the suite outdated and says industry-standard performance is approximately 100%. A score near 90% on a suite many tools now saturate says little about real-world testing.
• The tested path is narrower than the pitch. The README says the benchmarked path is a single agent, not the eight-operator swarm.
• Good signs, in fairness. The README reports an interval, separates black-box from white-box, marks a small CVE test as directional, and publishes data so a reader can check verdicts. Stripped transcripts limit how far an outsider can go.

What defenders should expect

Whatever the true score, the direction is clear and consistent with our master guide: open-source agentic testing is getting capable, can run on local models, and is aimed at the same targets whoever operates it. Expect faster and cheaper automated probing of web apps from testers and attackers alike. The difference defenders can control is the deployment: authorised and logged, or not. For threat context, compare ARTEX, the AntiFox fork and CAIRN; for a legitimate local environment, see Executioner.

What security leaders should do this week

• Write down who may run agentic testing. Require written authorisation and a scope list before an internal team points any agent at a system, and keep the record. The scoping checklist covers human-or-agent testers.
• Treat built-in scope limits as a convenience. An egress-scope feature inside the tool is not a control you own. Enforce scope with your own network rules and test accounts.
• Lock down coding-agent machines. Restrict egress and the credentials, tokens and cloud profiles a coding agent can read, because a tool like this runs with whatever the agent can reach. Practise in the agent egress containment lab.
• Log AI-agent tool calls. Record commands, network calls and file access by agents on endpoints and servers so automated testing from an internal host is visible and attributable.
• Detect automated probing. Tune web and API monitoring for rate, breadth and sequence of requests rather than single signatures; expect scans that adapt as they go.
• Review agent skills and plugins before running them. Add-ons and agent integrations execute with the agent's permissions; use the agent plugin supply-chain lab to set a review step.
• Keep dashboards private. If anyone runs a local-model setup or a browser console like this, bind it to localhost or a VPN and set credentials. Check your own ranges with the Open Agent Exposure Scanner, on hosts you own only.
• Read benchmark claims sceptically. When a vendor or project quotes a score, ask for the model, attempts, time limit and data. The agent benchmark explorer helps.
• Score the wider picture with the Agentic Attack Readiness Scorecard.

Frequently asked questions

What is T3MP3ST?

T3MP3ST is an open-source, AGPL-3.0 multi-agent offensive-security framework that its README calls an autonomous red teaming platform. It runs a recon, exploit and report chain from a browser "War Room" or a command line, using either a coding agent you already run or a local model. The README lists Claude Code, Codex, Hermes, OpenCode and Oh My Pi as connectable agents.

Did T3MP3ST really beat XBOW?

Its README says it scored a pass@1 mean of 90.1% on the 104-challenge XBEN suite in black-box mode and notes that XBOW self-reports 85% on the same suite. Both numbers are self-reported, and we found no independent reproduction. XBOW's 85% comes from an August 2024 post that names no model or attempt count, and the suite's own README now calls the benchmarks outdated. Treat "beat" as a claim, not a finding.

Are the two numbers comparable?

Not necessarily. Agent design, model, number of attempts, time limits, scoring and what the agent is told about the target all change a score. The T3MP3ST README reports pass@1 with a confidence interval and separates black-box from white-box runs; XBOW's post reports a single 85% and says the run was without human intervention. Different setups and different dates make a head-to-head reading weak.

Can the 90.1% be checked?

The README says every number recomputes from committed data and that a verification script re-derives the verdicts from files in the repository. That lets a reader check the graded results against the project's own data. It does not let an outsider re-run the agent, because the README says raw per-step transcripts are stripped. We did not run the script.

Does it need a cloud AI service?

No, per the README. It lists keyless local options (Ollama, LM Studio, vLLM and llama.cpp through an OpenAI-compatible server) as well as hosted providers. Local models mean a tester, or an attacker, does not need an account that a provider could suspend, which is one reason model-side abuse controls cannot be the only defence.

Is it safe or legal to use?

The README says it is for authorised testing, research and education, to be pointed only at systems you own or have explicit written permission to test, and that the user is solely responsible. It describes egress-scope containment on by default. Legality depends on your authorisation, not on the tool. We did not run it and cannot vouch for its safety.

What should defenders change?

Plan for faster, cheaper automated testing from anyone. Control authorisation and scope for any internal agentic testing, restrict egress and credentials on machines that run coding agents, log AI-agent tool calls, review agent skills and plugins before running them, and keep local dashboards off the internet.

Is there evidence of attackers using T3MP3ST?

We found none in the sources we read. The README and the coverage we found describe a tool for authorised testing. That is an absence of evidence in a few sources, not proof that it is unused, and the same could be said of any open-source offensive tool.

Sources

All opened 5 November 2026.

• elder-plinius/T3MP3ST README (scope, licence, supported agents, benchmark claims, authorised-use warning)
• XBOW: XBOW vs humans (5 August 2024; 85% of 104 benchmarks)
• XBOW: benchmarks post (104 benchmarks published; editor's note that they are outdated)
• xbow-engineering/validation-benchmarks README (104 benchmarks; outdated; industry performance approximately 100%)
• Original social-media post: not opened by us; not used as evidence.
• Searches for independent reproduction or news coverage found only the project's own pages and forks; no third-party verification of 90.1%.