AI Agent Observability Readiness Scorecard
Can you actually audit what your AI agents did from the telemetry you keep? This scorecard works through the controls that decide whether the answer is yes — framed around the Claude API's OpenTelemetry tracing (beta), which can emit spans for Messages requests, tool uses and Managed Agents sessions to any OTLP backend. Rate yourself across tracing enabled, coverage, retention for audit, content-mode governance, sensitive-data redaction, anomaly alerting and correlation & completeness. You get a banded 0–100 readiness score, a radar of where you are strong and weak, and the gaps ranked so you fix the highest-impact ones first. The weighting is a transparent heuristic, PlayCISO is not affiliated with Anthropic, nothing is uploaded, and the score is shareable by permalink.
You largely cannot see what your agents are doing. Either tracing is off or so little is captured that you could not reconstruct an incident. Start by turning on the Claude API OpenTelemetry tracing and exporting to a backend you control — everything else depends on it.
Score per dimension (0–100)
Prioritised gaps
Highest-weight, lowest-scoring controls first — spend remediation effort here.
- Tracing enabled · unanswered
Enable the Claude API OpenTelemetry tracing (beta): SDK tracing in code for the services you own, and/or API tracing in the Claude Console for organisation-wide coverage.
Reference: Claude API OTel tracing (beta)
- Tracing enabled · unanswered
Point the OTLP exporter at a tracing/observability backend you own, and confirm spans are actually arriving and queryable.
Reference: OTLP
- Coverage · unanswered
Confirm request-level spans (anthropic.messages.create, anthropic.model_request) appear for the features that call the model.
Reference: anthropic.messages.create
- Coverage · unanswered
Ensure every tool the agent can call emits a span (anthropic.custom_tool_use <tool>); untraced tools are untracked actions.
Reference: anthropic.custom_tool_use
- Retention for audit · unanswered
Set backend retention to match how late you realistically discover incidents; short default retention quietly erases the evidence.
- Retention for audit · unanswered
Keep raw, queryable spans (not just rolled-up metrics) so you can pull the exact trace for a specific user, feature or window.
- Content-mode governance · unanswered
Decide metadata_only vs content explicitly per environment, and record why; both are valid, but it must be a deliberate choice.
Reference: metadata_only vs content
- Content-mode governance · unanswered
Lock down who can read content-mode spans; captured prompts and responses are as sensitive as the data that went into them.
+ 16 more gaps in the full export.
Tracing enabled · weight 18
You cannot audit what you never recorded. The Claude API and SDK can emit OpenTelemetry spans (beta) to any OTLP backend — SDK tracing is set up in code, API tracing in the Claude Console (admin-only). Is any of it actually on and exporting?
OpenTelemetry tracing is turned on for your Claude API usage at all — SDK tracing enabled in code and/or API tracing enabled in the Claude Console — rather than running the models with no telemetry.
Spans are exported to an OTLP backend you control and can query — not emitted into a void or only printed to a local console.
If API tracing (Console-configured, admin-only) is used, the right admins know it is on and control who can turn it on or off.
Tracing is enabled in the environments that matter — production, not only a one-off developer experiment.
Coverage · weight 18
Which operations produce spans? Messages requests, each model run, client and server/custom tool use, and Managed Agents turns/sessions. Partial coverage leaves blind spots exactly where agent behaviour goes wrong.
Messages API requests are traced — anthropic.messages.create / anthropic.model_request spans are present for your calls.
Work inside a request is captured, including context compaction — anthropic.compaction spans — not just the single top-level call.
Tool use is traced — both client-side tools and server/custom tools (anthropic.custom_tool_use <tool> spans), which is where an agent most often does something unexpected.
Managed Agents turns and sessions are captured where you use them — anthropic.session.turn and anthropic.session.outcome_evaluation spans.
Retention for audit · weight 14
An incident is usually discovered after the fact. Traces have to survive long enough, and stay searchable, to reconstruct what an agent did weeks ago — not just power a live dashboard that rolls off in hours.
Traces are retained long enough to investigate an incident discovered weeks later, not expired within hours or a day.
Traces are searchable and queryable after the fact — by time, user, feature and trace id — not only as pre-aggregated charts.
Trace retention is a deliberate, documented policy aligned to your incident-response and audit needs.
Content-mode governance · weight 14
metadata_only (the default — ids, timing, status, token counts, names) vs content (adds the full prompts and responses). Turning on content gives you far richer audit trails and creates a new, highly sensitive data store. Choose, then govern it.
You have made a conscious choice of content mode — metadata_only (default) vs content — rather than leaving it to a default you never reviewed.
If content mode is on, access to spans carrying prompts and responses is restricted with least-privilege controls on the backend.
If content is captured, its retention on the backend is governed (shorter window or a separate store) rather than kept indefinitely beside the metadata.
Sensitive-data redaction · weight 14
Whatever ends up in a span leaves your trust boundary for the tracing backend. Secrets, credentials and PII must be kept out of exported prompts, responses, tool arguments and tool results — before export, not after a breach.
Secrets, keys and credentials are kept out of exported spans — in prompts, responses, tool arguments and tool results alike.
PII and regulated data are redacted or minimised before spans reach the tracing backend.
Tool-call arguments and results are reviewed for sensitive content, not assumed safe because the tool is "internal".
Anomaly detection & alerting · weight 12
Traces you never look at are just storage. The payoff is detection: prompt-injection patterns, tool misuse and unexpected egress, runaway agent loops, and cost or latency spikes — surfaced as alerts, not found during a post-mortem.
You alert on prompt-injection and jailbreak patterns — suspicious or adversarial instructions visible in traced activity.
You alert on tool misuse and unexpected egress — an agent or tool reaching something it should not.
You alert on runaway loops and abnormal turn counts in agent sessions.
You alert on cost and latency spikes using the token-count and timing data in spans.
Correlation & completeness · weight 10
A span is only useful if you can tie it back to a user, feature and conversation — and if you know its limits. Delivery is best-effort (spans can drop or duplicate; dedupe by trace+span id), and tracing is NOT a complete usage record.
Spans are linked to the user, feature and conversation — via gen_ai.conversation.id or your own trace/correlation id — so a trace maps back to who did what.
You dedupe spans by trace id + span id, because best-effort delivery can drop or duplicate them.
You understand tracing is best-effort and NOT a complete usage record — billing reconciliation and leaked-key / abuse detection use the Usage API, not the traces.
Educational self-assessment. The dimension weights are a transparent judgement framework for structured self-reflection, not a measurement, certification or guarantee of outcome. This tool is framed around the Claude API's OpenTelemetry tracing (beta); PlayCISO is not affiliated with Anthropic, and this is a security/audit self-assessment rather than a setup tutorial. It references span names and the Usage API by name only, quotes no invented figures, performs no scan, and runs entirely in your browser — nothing is uploaded. Your answers are stored only in the shareable link you choose to copy.
What this scorecard is
- A free, 100% in-browser self-assessment of one question: can you actually AUDIT your AI agents’ behaviour from your traces? Answer a checklist across seven weighted dimensions and get a banded 0–100 readiness score.
- Framed around the Claude API’s OpenTelemetry tracing (beta): the API and SDK can emit OTel spans for Messages requests, tool uses and Managed Agents sessions to any OTLP backend, using the GenAI semantic conventions.
- Seven dimensions: tracing enabled at all, coverage, retention for audit, content-mode governance, sensitive-data redaction, anomaly detection & alerting, and correlation & completeness.
- Every question maps to a concrete next action, and gaps are prioritised lowest-score, highest-weight first — so you spend effort where it most improves your ability to investigate an incident after the fact.
- The weighting is a transparent HEURISTIC for structured self-reflection, not a measurement or certification. PlayCISO is not affiliated with Anthropic. Nothing is uploaded — export to Markdown or CSV, and share the exact scorecard with a base64url permalink.
You cannot audit behaviour you never captured
When an agent does something it should not — follows an injected instruction, calls a tool it had no business calling, loops until it burns a fortune in tokens — the investigation only goes as far as your telemetry does. The Claude API and SDK can emit OpenTelemetry spans for Messages requests, individual model runs, client and server tool use, and Managed Agents turns and sessions, using the GenAI semantic conventions, to any OTLP backend. SDK tracing is set up in your code; API tracing is set up in the Claude Console and is an admin-level control. This scorecard turns “are we actually using any of that, and well enough to stand up in an audit?” into a structured checklist.
The subtleties are where readiness is won or lost. Capture has two content modes — metadata_only (ids, timing, status, token counts and names) and content (which adds the prompts and responses) — and the moment you turn on content you have created a new, sensitive data store that needs access control, retention and redaction. Delivery is best-effort: spans can be dropped or duplicated, so you dedupe by trace id and span id, and you treat tracing as rich behavioural evidence rather than a complete record — billing reconciliation and leaked-key or abuse detection belong to the Usage API. The seven weighted dimensions are a heuristic, shown openly so you can argue with them and tune them to your own environment. It is a self-assessment, not a certification, and it quotes no invented figures.
Frequently asked questions
What does the AI Agent Observability Readiness Scorecard measure?
It measures whether you could actually audit what your AI agents did from the telemetry you capture. It is organised around the Claude API’s OpenTelemetry tracing (beta), which can emit spans for Messages API requests, tool uses and Managed Agents sessions to any OTLP backend. You assess seven areas — is tracing even on, how much is covered, is it retained and searchable for after-the-fact investigation, have you governed metadata-only vs full-content capture, is sensitive data redacted before export, do you alert on anomalies, and are spans correlated to users and complete enough to rely on — and get a weighted 0–100 readiness score with a band and a prioritised gap list.
Is the score a real measurement or a benchmark?
No. It is a transparent heuristic framework for structured self-reflection, not a measurement, benchmark, audit or certification, and it predicts no outcome. The weights (tracing-on and coverage carry the most, because you cannot audit what you never recorded) are shown openly so you can adjust them to your own environment. The value is the structured conversation and the prioritised to-do list, not the exact number.
Does it scan anything, connect to Anthropic, or send my answers anywhere?
No. The scorecard runs entirely in your browser. It performs no scan, makes no API call, and touches no system. PlayCISO is not affiliated with Anthropic and this is not an Anthropic setup tutorial — it is a security/audit self-assessment that happens to be framed around the Claude API’s tracing feature. Your answers are never uploaded; they live only in the shareable permalink you choose to copy (encoded in the URL) and in any Markdown or CSV file you export yourself.
What is the difference between metadata_only and content mode, and why does the tool keep asking about it?
The Claude API’s OpenTelemetry tracing has two content modes. metadata_only (the default) records ids, timing, status, token counts and operation names — enough to see what happened without the words. content mode additionally captures the prompts and responses themselves. Content mode gives far richer audit trails but creates a new, highly sensitive data store, so the scorecard treats the choice, the access controls on those spans, and their retention as deliberate governance decisions rather than defaults.
If I turn tracing on, can I rely on the spans for billing or detecting a leaked key?
No, and the scorecard scores whether you understand this. Trace delivery is best-effort: spans can be dropped or duplicated, so you dedupe by trace id plus span id and never treat a span count as authoritative. Tracing is not a complete usage record — for billing reconciliation and for leaked-key or abuse detection, the Usage API is the source of truth. Traces are rich behavioural evidence; they are not a ledger.
This tool is free and always will be. Would a Pro workspace — saved designs, team sharing and exports — be worth paying for?
Show me Pro →No account needed to use this tool — this just tells us whether Pro is worth building out.