LLM Telemetry Privacy & Span-Redaction Checklist
Tracing your AI and agent calls is how you debug and govern them — but the moment you turn on full content, your spans can carry prompts, responses, tool arguments and tool results into a store under someone else's access and retention rules. Work through a checklist across seven controls that decide whether tracing stays an asset or becomes a new data-leak surface: defaulting to metadata-only, knowing which attributes carry content, redacting before export, backend access and retention, regulated-data handling, size and truncation limits and governance. You get a banded 0–100 score, a view of where you are strong and weak, and the gaps ranked so you fix the ones that matter most first. The weighting is a transparent heuristic, nothing is uploaded, and the result is shareable by permalink.
Trace content is at real risk of carrying secrets or personal data into a store you do not fully control. Start with the one control that stops a leak at source — redact before export — and keep content mode off until you can justify it.
Score per area (0–100)
Prioritised gaps
Highest-weight, lowest-scoring controls first — spend remediation effort here.
- Redaction before export · unanswered
Add an in-process span processor that redacts content attributes before export — the only place you fully control the data.
- Redaction before export · unanswered
Redact high-confidence secret patterns (API keys, tokens, credentials, private keys) from prompts, tool args and tool results before they are traced.
- Redaction before export · unanswered
Detect and minimise PII in the content attributes; free-text prompts and tool payloads are where it most often hides.
- Redaction before export · unanswered
Test the redaction against real spans, including nested tool-result structures; an untested scrubber is a false sense of safety.
- Default to metadata-only · unanswered
Keep metadata_only as the default and require a written reason before any destination is switched to content mode.
Reference: metadata_only (default)
- Backend access & retention · unanswered
Scope who can read trace content in the backend; prompt/response bodies should not be visible to every dashboard user.
- Backend access & retention · unanswered
Set an explicit retention window for trace content tied to its purpose, instead of accepting an indefinite default.
- Attribute awareness · unanswered
Document the five content-bearing attributes and treat each as a channel that can carry regulated or secret data when content mode is on.
Reference: gen_ai.* / anthropic.tool_result.content
+ 21 more gaps in the full export.
Default to metadata-only · weight 16
metadata_only is the default content mode and it is the safe one: ids, timing, status, token counts and names, but no prompts or responses. Turning on full content is a deliberate, justified, reviewed decision — not something you inherit by copying a config.
Tracing runs in metadata_only mode by default; full content mode is enabled only where a specific, written justification exists.
The team understands that metadata_only still carries token counts and names — including tool and MCP server names — and has checked those names are not themselves sensitive.
Content mode, where used, is scoped to the narrowest environment/destination that needs it (e.g. a debugging lane), not enabled globally across production.
Someone owns the tracing content-mode setting and reviews it on a cadence, so a temporary debugging enablement does not silently become permanent.
Attribute awareness · weight 14
If content mode is on, you must know exactly which span attributes carry prompts, responses, tool arguments and tool results — and have actually looked at what flows through them for your workloads.
The team knows the attributes that carry content: gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments and anthropic.tool_result.content.
You have reviewed real sample spans from your own workloads to see what actually lands in those attributes — not assumed from the docs.
System prompts are treated as sensitive: gen_ai.system_instructions can carry proprietary instructions, keys embedded in prompts, or customer context.
Tool arguments and tool results (gen_ai.tool.call.arguments, anthropic.tool_result.content) are recognised as a common path for PII and credentials to enter spans.
Redaction before export · weight 20
The one control that stops a leak at source: strip secrets and personal data from spans BEFORE they leave your process. Once a span is delivered to the backend, it is out of your hands.
A span processor / redaction step runs in-process and scrubs the content attributes before spans are exported.
Known secret shapes are redacted: API keys, bearer tokens, credentials, connection strings, private keys.
Personal data in prompts, tool arguments and tool results is redacted or minimised (names, emails, identifiers, free-text about people).
The redaction is tested — you have confirmed on real spans that secrets/PII are actually gone, including in nested tool-result JSON.
Redaction fails safe: if a span cannot be confidently scrubbed, content is dropped rather than exported raw.
Backend access & retention · weight 16
The observability backend stores whatever you send under ITS OWN access controls and retention. Trace content inherits the backend’s posture, not your app’s — so that posture has to be deliberate.
Access to trace content in the backend is restricted to named roles — not everyone with a login to the observability tool can read prompts and responses.
A deliberate retention period is set for trace content, matched to why you keep it — not an indefinite or vendor-default retention.
Trace content is in scope for your data-subject-access and deletion processes — you can find and remove a person’s data if it lands in spans.
You know where the backend stores data (region/tenancy) and that it meets your residency and contractual obligations.
Regulated-data handling · weight 14
HIPAA / FERPA / K-12 paths force metadata-only on API destinations regardless of the configured content mode. Special-category data should never ride in content spans — know which of your workloads are in scope.
You have identified which workloads touch regulated data (health, education/K-12, other special-category) and treat their tracing differently.
Regulated workloads are kept on metadata-only and you rely on the platform’s HIPAA/FERPA enforcement rather than assuming a content-mode flag protects you.
Special-category personal data is never relied upon to be safe inside content spans — the design keeps it out of traced attributes entirely.
The boundary between regulated and general workloads is enforced in configuration/routing, not left to developer discipline per request.
Size & truncation awareness · weight 8
Spans have a per-span content size limit (traces.max_content_bytes, ~0.75 MB default). Truncation is lossy and directional — don’t let it quietly drop the evidence you traced for, and don’t rely on traces for image/document bytes.
traces.max_content_bytes is set deliberately for your needs, not left unconsidered at the ~0.75 MB default.
The team knows truncation is directional — request spans drop the oldest messages / system prompt first, tool spans truncate from the end — and watches for anthropic.content.truncated=true.
You do not rely on traces to capture security-relevant content that truncation could silently remove (e.g. a long tool result).
The team knows images and documents appear in spans only as their type, not their bytes, and does not expect to recover them from traces.
Governance · weight 12
Tracing content is a data-processing activity. Record it, document the backend as a processor, and have a plan for the day the trace store itself is breached — because delivery is best-effort and the store concentrates prompts and results.
Exporting trace content is recorded as a data-processing activity (e.g. in your record of processing / DPIA where applicable).
The observability backend is documented as a processor / sub-processor with an appropriate agreement in place.
There is an incident plan for a breach of the trace store itself — it concentrates prompts, responses and tool results in one place.
The team treats delivery as best-effort and does not depend on traces as the system of record for compliance or security evidence.
Educational self-assessment, not legal advice. The weights are a transparent judgement framework for structured self-reflection, not a measurement, certification or guarantee of outcome. The capability facts describe Claude's API OpenTelemetry tracing (a beta capability) and are provided for education only; PlayCISO is not affiliated with Anthropic. This tool performs no scan and runs entirely in your browser — nothing is uploaded. Your answers are stored only in the shareable link you choose to copy.
What this checklist is
- A free, 100% in-browser self-assessment for teams adding OpenTelemetry-style tracing to AI/agent systems: instrument your LLM calls WITHOUT turning your observability backend into a new data-leak surface.
- Seven weighted areas: default to metadata-only, attribute awareness, redaction before export, backend access & retention, regulated-data handling, size & truncation awareness, and governance.
- Every item maps to a concrete next action, and the gaps are prioritised lowest-score, highest-weight first — so the one control that stops a leak at source (redaction before export) gets attention first.
- Built on accurate facts about Claude’s API OpenTelemetry tracing (a beta capability): metadata_only is the default content mode; full content mode adds prompts and responses carried by specific span attributes. PlayCISO is not affiliated with Anthropic.
- Educational self-assessment, not legal advice. Nothing is uploaded — export to Markdown or CSV, and share the exact checklist with a base64url permalink.
Observability should not become your biggest data store of prompts
An AI trace is uniquely dense: in one place it can hold the system prompt, the user's request, the model's response, the arguments your tools received and the results they returned. That is exactly what makes tracing useful for debugging an agent — and exactly what makes the trace store a tempting target. The safe default is metadata-only: ids, timing, status, token counts and names, but no bodies. Turning on full content should be a deliberate, justified decision, scoped to where you actually need it, with the content-bearing attributes understood and a redaction step that runs before spans leave your process. Once a span is delivered, it lives under the backend's controls, and delivery is best-effort.
The seven areas are weighted on purpose — redaction before export carries the most, because it is the one control that stops a secret or a piece of personal data reaching the trace store at all, followed by defaulting to metadata-only and knowing which attributes carry content. That weighting is a heuristic, shown openly so you can argue with it and adjust it to your own environment. The capability facts the checklist is built on describe Claude's API OpenTelemetry tracing, a beta capability; they are represented here for education only, and PlayCISO is not affiliated with Anthropic. This is an educational self-assessment, not legal advice or a certification, and it quotes no invented statistics.
Frequently asked questions
What does this checklist assess?
It is a pre-deployment self-assessment of the controls that keep LLM and agent tracing from leaking sensitive data. You answer items across seven weighted areas — defaulting to metadata-only, knowing which span attributes carry content, redacting secrets and personal data before spans leave your process, controlling backend access and retention, handling regulated data, understanding size and truncation limits, and governing the whole flow — and get a weighted 0-100 score with a band and a prioritised gap list.
What are the two content modes in Claude’s OpenTelemetry tracing?
metadata_only is the default: it captures ids, timing, status, token counts and names such as tool and MCP server names, but no prompt or response bodies. content mode adds the model prompts AND responses. When content mode is on, the sensitive content is carried by specific attributes: gen_ai.system_instructions (the system prompt), gen_ai.input.messages (the request), gen_ai.output.messages (the response), gen_ai.tool.call.arguments (tool arguments) and anthropic.tool_result.content (tool results). This tool represents these facts for education only; PlayCISO is not affiliated with Anthropic.
Why is redaction before export the highest-weighted area?
Because it is the one control you fully own. Once a span is delivered to the observability backend, the content is stored under the backend’s access and retention settings and is out of your hands. An in-process span processor that strips API keys, tokens, credentials and personal data from the content attributes before export is the only place you can reliably stop a leak at source — and it should fail safe, dropping content it cannot confidently clean rather than exporting it raw.
What about regulated data like health or education records?
HIPAA, FERPA and K-12 restrictions force metadata-only on API destinations regardless of the content mode you configure. The checklist asks you to identify which workloads touch regulated or special-category data and to keep them on metadata-only by design, rather than relying on a content-mode flag or on redaction to catch everything. Special-category data should never be relied upon to be safe inside content spans.
How do size limits and truncation affect what I can trust in a trace?
There is a per-span content size limit, traces.max_content_bytes, defaulting to about 0.75 MB. Truncation is directional: request spans drop the oldest messages and system prompt first, while tool spans truncate from the end, and a truncated span is marked anthropic.content.truncated=true. So a trace is not a complete audit log — truncation can silently remove the exact content you needed. Images and documents appear only as their type, not their bytes, so you cannot recover them from traces.
Does it scan anything or send my answers anywhere?
No. The checklist runs entirely in your browser. It performs no scan and touches no system. Your answers are never uploaded — they live only in the shareable permalink you choose to copy (encoded in the URL) and in any Markdown or CSV file you export yourself.
This tool is free and always will be. Would a Pro workspace — saved designs, team sharing and exports — be worth paying for?
Show me Pro →No account needed to use this tool — this just tells us whether Pro is worth building out.