Agent Trace Triage
A queue of Claude-agent traces lands in your observability tool — a weather lookup, a ticket summary, a web-research run, a batch job, a session that seems to “forget” its instructions. Each is a rendered OpenTelemetry span tree: an anthropic.messages.create request and its model_request and custom_tool_use children, with a few attributes showing. Some are perfectly benign. Some hide a prompt injection, a tool reaching outside its task, data leaving to an external host, a token-burning runaway loop, or just a truncation that looks worse than it is. For each, call the verdict class and name the exact attribute that gives it away — gen_ai.input.messages, anthropic.tool_result.content, gen_ai.tool.call.arguments or the span/token counts. Read the attributes, not the vibe. Defensive only; synthetic traces, no working exploits.
The real job. More noise, a tighter clock.
Trace queue
1 / 10 triaged
User asks for the weather in Denver
A single-turn chat that calls one read-only tool and answers.
OTel span tree
Tip: inspect the highlighted attribute before you decide — the span names alone rarely tell you which class it is.
1 · Verdict
2 · Primary tell (the attribute)
Pick a verdict and the primary tell to continue.
Turn & token budget
This trace emitted 2 model_request spans at ~1,180 input tokens. One signal among several — weigh it with the messages, the tool args and any tool result.
Triage
A queue of 10 Claude-agent OpenTelemetry traces. For each, call the verdict class — benign vs prompt injection, tool abuse, exfiltration, runaway loop or a benign misconfig — and name the primary tell (the attribute). Several are benign — do not over-flag.
So far: 0 / 10 traces triaged.
Live arena
4 activeadversary staging…
What this lab is
- A free, browser-based triage drill: work a queue of realistic Claude-agent OpenTelemetry traces and, for each, call the verdict class (benign vs attack) and name the exact attribute that gives it away.
- Uses real Claude OTel span names — anthropic.messages.create, anthropic.model_request, anthropic.custom_tool_use, anthropic.session.turn, anthropic.compaction — and GenAI-convention attributes like gen_ai.input.messages, gen_ai.tool.call.arguments, anthropic.tool_result.content, gen_ai.usage.input_tokens and anthropic.content.truncated.
- Six verdict classes: benign, prompt injection, tool abuse / excessive scope, data exfiltration, runaway loop / cost blow-up, and misconfig / benign anomaly. Each has a specific tell — an injected instruction in the input or a tool result, an over-scoped or external-destination argument, a span/token spike, or a truncation / fallback flag.
- Several traces are benign on purpose — including a heavy, high-token batch that only looks like a loop — so the skill is telling a real incident apart from busy-but-fine activity without over-flagging.
- Every trace is a synthetic, illustrative example; payloads are defanged and no statistics are invented. Claude OTel telemetry is a beta feature represented here accurately — PlayCISO is not affiliated with Anthropic. Difficulty tiers, a speed bonus and an S–D grade.
The verdict follows the attributes, not the shape of the trace
Agent telemetry is powerful precisely because the attack usually leaves a mark in a specific attribute. A direct prompt injection shows up as an override string inside gen_ai.input.messages; an indirect one rides in on anthropic.tool_result.content — a fetched page or ticket that tells the model to do something, and the model obeys. Tool abuse and exfiltration both live in gen_ai.tool.call.arguments: the difference is whether the over-scoped argument stays local (reading /etc/passwd, a wildcard secret read) or ships data to an outside host. A runaway loop is a pile of near-identical anthropic.model_request / anthropic.session.turn spans with gen_ai.usage.input_tokens climbing and no progress.
Just as important: do not over-flag. A high-token, many-turn batch job looks exactly like a loop until you notice each turn processes a different item and makes progress — a loop repeats, a batch advances. A confusingly wrong answer looks like a jailbreak until anthropic.content.truncated=true or an anthropic.compaction span explains that the model simply lost part of its context, and a sudden quality drop can just be a fallback model in anthropic.model_request. Those are reliability bugs, not incidents, and escalating them as attacks burns response time. No single span count or scary-looking field is the verdict — triage is weighing the attributes together and spending attention where the evidence points.
Frequently asked questions
What does the Agent Trace Triage lab teach?
It trains the fast call a SOC analyst or platform engineer makes when a Claude agent's OpenTelemetry trace lands in front of them: is this benign activity or an incident, and which one — prompt injection, tool abuse / excessive scope, data exfiltration, a runaway loop, or a benign misconfiguration? You work a queue of rendered span trees (a Messages request and its model_request / tool_use children, with a few attributes shown) and for each you pick the verdict class and the primary tell — the exact attribute that proves it. It is detection and triage practice; it does not read your real traces.
Are these real Claude OpenTelemetry spans?
The span and attribute names are real. Anthropic exposes agent telemetry through OpenTelemetry using span names such as anthropic.messages.create (emitted CLIENT-side by the SDK and SERVER-side by the API), anthropic.model_request, anthropic.custom_tool_use <tool_name>, anthropic.session.turn and anthropic.compaction, with attributes following the OpenTelemetry GenAI semantic conventions — gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments, anthropic.tool_result.content, gen_ai.conversation.id, anthropic.session.id, gen_ai.usage.input_tokens and anthropic.content.truncated. This telemetry is a beta feature; the lab represents it accurately as a defender exercise. The traces themselves are synthetic illustrative scenarios, not captured data, and PlayCISO is not affiliated with Anthropic.
Does it attack an agent or generate exploits?
No. This is strictly a defensive training drill. Every trace is a synthetic, illustrative example and every payload is defanged — injected instructions are quoted as text, external hosts are written as evil-sink[.]io, and no real secrets, tokens or working endpoints appear. Nothing here can be run against a live agent; it only trains recognition of the patterns in telemetry you already collect.
What are the real-world mitigations it points to?
Each class maps to a concrete control. Prompt injection (in gen_ai.input.messages or anthropic.tool_result.content): treat tool output as untrusted data, keep durable constraints in system instructions, and compare the system prompt against what the input tries to force. Tool abuse / exfiltration (gen_ai.tool.call.arguments): scope tools to explicit arguments and allow-listed paths and destinations, and alert on external hosts in tool args. Runaway loops (repeated model_request / session.turn spans and a climbing gen_ai.usage.input_tokens): set iteration and retry caps and token budgets. Benign anomalies (anthropic.content.truncated=true, a fallback model, compaction dropping early turns): fix routing, chunk inputs and pin critical constraints — these are reliability issues, not incidents, and mis-escalating them wastes response time.
This tool is free and always will be. Would a Pro workspace — saved designs, team sharing and exports — be worth paying for?
Show me Pro →No account needed to use this tool — this just tells us whether Pro is worth building out.