Access to Production LLM Traces by Role

A production LLM trace almost always contains more than the person viewing it needs. It may include system instructions, user text, fragments of RAG context, tool calls, arguments, responses, customer identifiers, and quality scores. If you grant access to such a record on the basis of «can view traces», you have already combined several different permissions into one.

A workable approach starts neither with the choice of storage nor with IAM checkboxes. First, break the trace down into data types and decide which role needs to solve which task. A developer must quickly find a regression. Support must investigate a specific customer case. An auditor must verify that the rules are being enforced. None of these tasks, by itself, requires permanent access to the entire original conversation.

A trace is not a single access object

A trace looks like one record in an observability interface, but from an access-control perspective it is a set of objects with different levels of risk. As long as a team stores it as a monolithic JSON object, it is forced to choose between two bad options: broad read access or withholding the data people need to do their work.

Split the contents into at least four layers.

• Operational metadata: trace_id, time, service, environment, tenant, route, application version, model, provider, latency, error codes, completion reasons, input and output tokens, and cost or its internal equivalent.
• Request content: system instructions, user messages, attachments, retrieved documents, tool arguments, and parameters received from the application.
• Response content: model text, structured responses, tool-call arguments, tool results, and refusal messages.
• Derived artifacts: quality scores, error classes, security-check results, PII signals, hashes, template fingerprints, short excerpts, and aggregates.

OpenTelemetry makes this distinction visible in its GenAI semantic conventions. Its documentation separately identifies retrieved documents, system instructions, arguments, and tool results as potentially sensitive. It also says that attributes that may contain sensitive data should be opt-in rather than collected by default. This is a useful engineering boundary: collect useful metadata consistently, and enable original content deliberately under a policy.

Do not confuse two operations. Collection determines what enters storage. Visibility determines what a particular person receives when reading it. If you collect the full text of every transaction and then hope to hide it in the UI, the secret has already passed through the collector, queue, indexes, backups, and possibly an export. A screen mask does not cancel that route.

The access matrix must describe actions and fields

The role «viewer» is useless unless it answers two questions: which fields can a person see, and what can they do with them? The access matrix must be checked for search, opening, export, API reads, saved queries, notifications, and attachment downloads. Teams often protect only the details page while a CSV export or internal endpoint returns the original payload.

Here is a starting matrix. It does not replace data classification, but it clearly shows why a general read role breaks down within the first month of operation.

This does not mean a developer should never see text. It means text should not be the price of routine debugging. The developer first sees enough to say: the error appeared in version 2026.07.3, affected the claims-summary route, occurs only with one provider, happens after a tool call, and is linked to a specific template. If that is not enough, they request disclosure of one trace or a short set of related traces.

Support works differently. It rarely needs a statistical sample of all customers. It needs one context tied to a case and enough information to explain what happened. Its main restriction should therefore apply not only to fields but also to the record set: a specific tenant, period, ticket, or external case ID. A search such as «find all messages containing ...» usually creates unnecessary risk for support and offers little practical value.

An auditor checks the control, not the literary quality of the model's response. What matters is the immutability of events, the history of policy changes, proof that masking was applied, the log of temporary disclosures, and the link between the trace and the application version. Let the auditor see that a field existed, was classified, was hidden by policy, and was not exported. The original text is needed only in exceptional procedures, if it is needed at all.

Metadata provides most of the diagnosis

Most LLM incidents can be narrowed down without the full prompt. Teams often fail to do this simply because recording text is convenient, not because the text is necessary.

The following signals should be available for the initial investigation:

• trace, parent span, and correlation IDs for the application request;
• a tenant identifier in the form allowed for the role;
• model, provider, route, execution region, template version, and code version;
• total request duration, time to first token, token counts, stop reason, and tool-call status;
• error type, response code, retry count, whether fallback was used, and why the switch occurred;
• content classes: PII detected, attachment present, search result present, tool called, score below threshold.

OpenTelemetry recommends recording only the details genuinely needed for a particular operation, rather than every available property. Its documentation explicitly says that unnecessary attributes increase telemetry volume and overhead, and that when in doubt it is better not to add an attribute until a confirmed need appears. For LLM traces, this is also a security rule.

Metadata becomes more useful when it is normalized. Do not put the model name only in free-form log text. Do not make the template name part of an arbitrary message. Do not record error codes as long exceptions that may include input data. Fields should be separate, use stable names, and have a limited set of values.

For example, instead of this event:

{
  "message": "Retrying gpt request for client acme: prompt failed after tool returned passport 123456789"
}

record an event that is searchable and safe for a broad group of engineers:

{
  "trace_id": "01JQ4F9B6XZ3R4H8V0M2K7C1P9",
  "tenant_ref": "t_7a91",
  "service": "claims-api",
  "route": "POST /v1/claim-summary",
  "prompt_template_version": "claims-v18",
  "model": "provider/model-name",
  "attempt": 2,
  "fallback_used": true,
  "error_class": "tool_result_schema_error",
  "pii_detected": true,
  "input_tokens": 1834,
  "output_tokens": 0
}

This set answers the questions that arise in the first ten minutes: where is the failure, who is affected, did the version change, was fallback used, and is there a risk of sensitive data? A substantive investigation can then request one trace_id rather than search across every conversation.

Masking must preserve diagnostic meaning

Masking that turns every text into [REDACTED] looks safe in a demo and is useless during a real shift. An engineer cannot distinguish empty retrieval from a wrong instruction, and a support employee cannot tell that a customer sent a file instead of text. But a mask that leaves too much becomes decorative control.

Preserve the shape, not the content. Instead of the original phrase, show the message type, length, class of detected entities, language, hash of the normalized fragment, and a short safe preview. Build the preview after removing detected PII and secrets, not before.

Example of a useful message projection:

{
  "role": "user",
  "chars": 842,
  "language": "ru",
  "content_hash": "sha256:9ce1...aa84",
  "pii_types": ["person_name", "phone"],
  "secret_types": [],
  "preview": "User asks for an update on case [PERSON] by number [PHONE]",
  "attachment_count": 1
}

A hash does not make text safe by itself. If the possible value space is small, it can be brute-forced. Do not hash contract numbers, national ID numbers, email addresses, and other short identifiers separately and assume that is enough. A hash is useful for comparing duplicates and deduplicating long messages when access to the original is already blocked. For a sensitive identifier, store a separate internal reference that has no meaning outside the protected environment.

Be especially careful with tool arguments. They often contain an account number, address, customer search result, or internal API payload. OpenTelemetry separately warns about the sensitivity of gen_ai.tool.call.arguments and gen_ai.tool.call.result. In an ordinary trace, show the tool name, result schema, status, duration, and field classification. Store original arguments separately and disclose them only by exception.

In its Sensitive Information Disclosure section, OWASP treats PII, financial information, health data, credentials, and confidential documents as sensitive information that an LLM application may expose. It also warns that a prohibition in the system prompt is not reliable protection because prompt injection and other paths can bypass it. Trace policy must not depend on the model's obedience.

Store full text separately from the search index

The rule is simple: if people can search the text without special permission, the text is already too widely available. Do not index original prompts and responses in the same search environment where engineers look for errors, latency, and costs.

You need two representations of one trace. The first, operational, contains metadata and masked projections. It is available to most technical roles and optimized for filtering. The second, protected, contains encrypted original segments linked to trace_id and a specific field. It is not used for full-text search and is not included in batch exports.

A record structure might look like this:

trace_envelope
  trace_id
  tenant_ref
  timestamps
  operational_metadata
  safety_labels
  masked_message_views
  content_refs[]

content_vault
  content_ref
  trace_id
  field_path
  encrypted_payload
  retention_class
  access_policy_version
  content_hash

trace_envelope answers everyday questions. content_vault exists for investigations that can be shown to be necessary. Do not duplicate encrypted_payload in error queues, dead letter queues, the analytics lakehouse, and database snapshots. When a team builds four copies «just in case», access to the original becomes impossible to explain even to the team itself.

The policy must be applied on read, not only on write. There are two reasons. First, a trace's classification may change after a more accurate scan. Second, a person's permissions and request context change. Support may have opened a record under an active ticket in the morning, while the ticket is already closed in the evening. The read service must evaluate the role, tenant, purpose, object, time, and approval immediately before serving the content.

Row filters and column masks illustrate the required property well: restrictions must take effect before the user sees the original value. Databricks documentation describes row filters as restricting rows during a query and column masks as substituting a masked value at read time. It also notes that when optimization conflicts with protection against disclosure, the engine chooses protection. That is the right priority order for LLM traces.

Developers need a disclosure path, not a permanent privilege

Banning full text forever is easy. Then a defect appears that reproduces only on one rare input, and someone gives a group administrator rights «until the investigation is over». A permanent gap appears because nobody removes the temporary permissions.

Make disclosure a normal engineering operation. It should be fast enough for a real incident and inconvenient enough to deter curiosity.

1. The engineer specifies a trace_id or a limited set of related traces, the reason, the ticket, and the expected access duration.
2. The system checks whether the answer can be obtained without the original text. If so, it returns the request with a masked view and the necessary metadata.
3. Full content requires approval from the service owner, the security on-call person, or another designated approver. Do not assign approval to the requester.
4. Access is short-lived, does not provide bulk search, and does not enable export. After expiration, the read service returns the masked projection again.
5. Every read creates a separate audit event, even when one approval covers several views.

Separate the «right to request disclosure» from the «right to read the original». Any engineer responsible for the service may have the first. The second appears only after the context has been evaluated. This preserves speed without turning every developer into a privileged storage operator.

Do not grant access based on a team name without a data scope. A payments-service developer should not be able to open traces from a medical chatbot simply because both belong to Engineering. The policy input should include at least the role, service, tenant, data classification, access purpose, and approval status.

Support should see the case history, not the entire tenant

Support faces a different kind of pressure. A customer writes: «Yesterday the assistant disclosed the wrong document. Please check.» The employee needs to understand the sequence, but broad searches across customer messages would create a new leak channel.

Tie support access to the case identifier. The system can find a trace by ticket, external request ID, time, a protected CRM reference for the user, or a session ID if policy allows. It then returns a limited timeline: masked messages, call statuses, version numbers, scores, whether a document was used, and the final response in a safe projection.

A useful support card answers specific questions:

• did the model return a response, or was there a technical failure;
• did the security policy trigger, and why;
• was RAG used, how many documents were returned, and is there an indication of empty context;
• was a tool called, and did it end in an error;
• did the model, template, or route change between the successful and unsuccessful cases.

Do not add a «show all customer messages» button. If content is necessary to answer a complaint, support submits a disclosure request with the case number. In some organizations, the data owner, rather than the support manager, must approve this request. The category of data determines the details, but the principle is the same: an employee must not decide that curiosity is equivalent to a business need.

Tenant restrictions must work before search. If the service searches the entire database first and hides other tenants' rows in the interface afterward, risks remain in search suggestions, result counts, caches, and API errors. The tenant identifier must be part of the predicate at the storage-query level.

Audits prove control without reading conversations

Audits often receive either too much or almost nothing. In the first case, auditors see original conversations they do not need. In the second, they receive a polished policy document with no way to check how it was applied. Both approaches are weak.

An auditor needs a time-linked set of evidence:

• the access matrix version and the date it changed;
• a list of roles, groups, and areas of responsibility;
• events for granting, denying, extending, and revoking temporary access;
• events for reading protected content, with the subject, object, reason, and policy decision;
• hashes or signed manifests to verify trace integrity;
• results from regular tests confirming masking and tenant scoping.

Immutability does not mean that nothing is ever deleted. Content may have a short retention period under the data policy. After deletion, retain a minimal record of the fact itself: trace_id, time, data category, retention-policy version, deletion-operation ID, and a cryptographic fingerprint if retaining it is permitted. The auditor can then see that the record existed and was deleted under a rule, without receiving the text.

In NIST SP 800-53, the AC-6 family requires access to security functions and related security-relevant information to be explicitly authorized under the principle of least privilege. For production traces, this means that access to the ordinary observability panel must not automatically include the ability to read original messages.

Test bypass paths, not just the policy. Create a synthetic trace containing a name, a phone-like string, a secret-like token, a tool argument, and an error response. Then run the same requests as a developer, support employee, and auditor through the UI, API, export, saved search, and notification mechanisms. Compare the actual payload with the expected matrix. If one endpoint returns the original, you do not have access control, even if the main page looks tidy.

Retention and quality evaluation need separate policies

Teams often keep prompts «for evals» even though they need not the prompts themselves but a reproducible way to understand why a score dropped. These are different tasks.

For regular quality evaluation, it is useful to retain derived data: dataset ID, prompt version, model version, scenario, rubric, scores, failure reasons, input hash, masked projection, and a link to protected content. This is enough to build trends, compare routes, and find regressions. The original is opened only by experts labeling a disputed case or investigating a specific discrepancy.

Do not use one retention period for everything. Latency and error metadata may be needed longer than raw conversation text. Scores may outlive the original content if they do not allow it to be reconstructed. Attachments and tool results often require an even shorter period because they carry data from external systems.

For teams that need local storage, audit logs, PII masking, and key-level restrictions, AI Router can be one of the environments where this access model is enforced alongside LLM request routing. But the matrix itself must live in the application architecture and be checked independently of the selected gateway.

Start with one real trace that is currently visible to too many people. Break it down into metadata, original content, and derived artifacts. Then ask a developer, a support employee, and an auditor to formulate three questions each that they need to answer. Anything that does not help answer those questions should not be part of their permanent access.