Model Audit
Adversarial probe suite against a model endpoint you control.
The problem
Before each release you should know: can it be jailbroken, will it leak its system prompt, does it follow injected instructions, will it write malware? Manual red-teaming does not scale.
What it does
Point it at an OpenAI- or Anthropic-compatible endpoint you own. It runs a fixed adversarial suite — jailbreak, harmful-content, prompt-injection, and secret-leak probes — evaluates each reply with transparent heuristics, and returns the model’s own responses so you can verify. Your key is used only for the run.
Capabilities
- Jailbreak, harmful-content, injection & canary-leak probes
- Resilience score (0-100) with per-probe pass/fail + rationale
- Returns the model’s raw replies for your own review
- OpenAI- and Anthropic-compatible; SSRF-guarded
- Runs in the browser or as an npm package / CI gate
How you run it
Enter your endpoint, key, and model in the browser and run the suite live. Subscribers can install @playciso/modelaudit and fail CI if any probe regresses.
Roadmap
- Adversarial probe suite + scoring
- In-browser live runner (OpenAI/Anthropic)
- npm package + CI gate
- LLM-judge scoring + bias/drift axes
- Baseline diffing across releases
Model Audit runs inside PlayCISO for subscribers. The source stays private — no public repos, nothing to fork, nothing for attackers to study. Weekly, monthly, and yearly plans all include every tool.