You Can Finally Trace Your AI Agents: Claude’s OpenTelemetry Export and the Audit Evidence It Gives You

For two years the honest answer to "what did your AI agent actually do?" was a shrug. You had the final output, maybe a log line, and little in between. That changes with tracing. Anthropic’s documentation (marked beta) describes OpenTelemetry export for the Claude API and SDK: you can now trace Messages API requests, tool uses and Managed Agents sessions as OpenTelemetry spans and ship them to any OTLP observability backend. For a security or audit leader that is not a developer-convenience footnote — it is the evidence layer an AI audit has been missing.

Independent, docs-grounded analysis for security & audit leaders. PlayCISO is not affiliated with Anthropic; every feature detail below is drawn from Anthropic’s published OpenTelemetry documentation (beta) and relayed with attribution. Span and attribute names are quoted from those docs.

What it traces — the parts that matter for audit

The spans map almost one-to-one onto the questions an auditor or incident responder asks. A request shows up as anthropic.messages.create (the SDK client span and its API server span share the name — you tell them apart by span kind), with each model run as anthropic.model_request. Tool calls surface as anthropic.custom_tool_use <tool_name> — for example anthropic.custom_tool_use list_issues — carrying gen_ai.tool.call.arguments and the result in anthropic.tool_result.content. Managed Agents add anthropic.session.turn and anthropic.session.outcome_evaluation, and a whole session groups by anthropic.session.id. Token usage rides along in gen_ai.usage.input_tokens, with compaction tracked separately.

Put together, that is a reconstructable account of what the agent was told (gen_ai.system_instructions, gen_ai.input.messages), what it decided to do (the tool spans and their arguments), what it got back, and what it produced — the raw material of an AI audit trail.

Why this is the evidence layer your AI audit needs

Every "how do we audit AI?" framework — ISO/IEC 42001, the NIST AI RMF, the EU AI Act’s logging and record-keeping duties — assumes you can produce evidence of how a system behaved. For traditional software you have logs; for agents, most teams had almost nothing between prompt and answer. Tracing closes that gap. It also powers detection: because you can see tool calls and their arguments, you can alert on the agent reading something it shouldn’t, calling a tool the task never required, sending data to an unexpected destination, or looping until the token bill spikes. Observability stops being a performance concern and becomes a security control.

The new surface: your traces can become your biggest data leak

Here is the part a CISO must not miss. In metadata_only mode (the default) spans carry identifiers, timing, status, token counts and names — low-sensitivity, safe to export broadly. Switch to full "content" mode and the spans now contain the actual prompts and responses: system instructions, user messages (with whatever PII, credentials or confidential text a user pasted), tool arguments and tool results. Those spans flow into your observability platform and live there under its access controls and retention — which may be far looser than the controls around your application database. You have effectively created a second copy of your most sensitive AI data in a monitoring tool.

Anthropic’s docs give you levers: a per-span content size limit (around 0.75 MB, with oldest messages and tool-result tails truncated first and an anthropic.content.truncated flag), and forced metadata-only export for HIPAA/FERPA/K-12 destinations. But the governance is yours: redact secrets and PII before spans leave your process, lock down and time-box who can read trace content, treat the trace store as an in-scope data processor for DSARs and deletion, and only enable full content where that’s all true.

Three honest limits

• It’s beta. Anthropic provides the export under the Beta Services terms; expect change.
• Delivery is best-effort. Spans can be dropped and retries can duplicate them — dedupe by trace id + span id, and never treat traces as a complete record.
• It is not your billing source. For authoritative per-key usage and leaked-key detection, Anthropic points to the Usage API; use traces for debugging, detection and audit context, not accounting.

Put it to work

We built three free, in-browser tools around exactly these decisions:

• AI Agent Observability Readiness Scorecard — can you actually audit agent behaviour from your traces? Score coverage, retention, content-mode governance, redaction and alerting.
• Agent Trace Triage Lab — read realistic Claude OTel spans and call it: benign, prompt injection, tool abuse, exfiltration or a runaway loop — and the tell.
• Telemetry Privacy & Span-Redaction Checklist — instrument tracing without turning your observability stack into a new data-leak surface.

And to place it in the bigger picture, see the AI Audit & Assurance Readiness tool and the EU AI Act Classifier — traces are the evidence those audits run on.

The takeaway for leaders

Tracing finally makes agent behaviour auditable — adopt it, because "we couldn’t see what the agent did" is no longer an acceptable answer to a board or a regulator. Start in metadata-only, wire up detection on tool calls and sessions, and gate full-content tracing behind the same data governance you’d demand of any system holding prompts and PII. The visibility is the win; the discipline is making sure the visibility doesn’t become the breach.

Source

Based on Anthropic’s published documentation; PlayCISO originates none of the feature details and is not affiliated with Anthropic.

• Anthropic — OpenTelemetry for the Claude API (beta)