Your agent's debug logs are another data export

Prompt capture, tool arguments and exception messages can copy private records into telemetry. How to test observability without treating missing content as a clean run.

Published
Updated
Reading time
7 minutes
#whattheexporterreceives+--------------------------++--------------------------+|redactedbeforeexport||debugcapturelefton||||||opcustomer_lookup||opcustomer_lookup||statustimeout||statustimeout||errorUpstreamTimeout||errorUpstreamTimeout||args[removed]||args################||||||kept:whatoperationsneed|exported:thepayload|+--------------------------++--------------------------+app---->collector---->logbackendthebackendisarecipient,evenwhenonlyemployeescanopenit

A tool call fails and the exception handler prints its arguments. The user sees a short error message. The logging service receives the email address, internal note and credential embedded in the failed request. Nothing in the final answer reveals that second export.

Give telemetry a place in the data flow

Draw the route from the application to the local logger, collector, storage backend and any alerting integration. At each hop, name the fields and the readers. The OWASP Logging Cheat Sheet advises excluding or protecting credentials and sensitive personal data, restricting log access and controlling retention. Apply those checks to agent logs just as you would to other application logs.

A useful operational record can contain a run identifier, operation name, duration, status and error class. It often does not need the whole prompt or tool response. Decide field by field rather than relying on a single switch called debug.

Check what your instrumentation captures

The OpenTelemetry GenAI attribute registry treats message content as potentially sensitive and recommends opt-in capture for it. Verify the behavior of the instrumentation and version you deploy; a convention does not enforce a library's configuration. Framework callbacks, custom log statements and exception reporters may each export additional content.

Disabling prompt capture reduces that export, but changes the evidence available to a privacy audit. A trace containing only timing and token counts cannot tell you whether a missing tool argument held a secret. Record that coverage gap explicitly.

Test the exporter with synthetic records

Put a distinctive marker in a fake tool response. Exercise a success, a timeout, an invalid response and a retry. Inspect the records presented to each exporter, including exception bodies and span events. If an error reporter has a separate transport, test it separately.

Make the intended output concrete. This illustrative application record retains the operational fields while omitting the private payload:

{
  "run_id": "synthetic-log-test-01",
  "operation": "customer_lookup",
  "status": "timeout",
  "error_type": "UpstreamTimeout",
  "attempt": 2
}

Then deliberately inject the marker into one test log record. Confirm the recorder captures it and the detector reports it. Remove that injected disclosure and confirm the ordinary logging path stays free of the marker. This positive control checks that a passing result is meaningful.

Redact before the forbidden destination

OpenTelemetry's sensitive-data guidance describes filtering and transforming telemetry. Choose the location by the permission boundary. If raw data must not leave the application, a filter in a remote collector acts too late. If an approved local collector is the boundary, inspect its outgoing records as well as its configuration.

Keep correlations that serve debugging, but review them as data too. User identifiers, file paths and exception text can reveal information after the obvious payload is removed. Test nested fields and alternate exporters, since a redacted main stream does not cover a second copy.

Reasoning traces are logs too

The AgentLeak paper measured logs as channel C6 and found them leaking more often than tool calls. It calls the difference the thought gap: an agent works through sensitive data in its reasoning, the reasoning is written to a log, and the agent then decides not to pass the data on. The tool call is clean and the log is not. For Claude 3.5 Sonnet the study measured 74% on tool calls and 85% on logs. Across all models, in more than 65% of scenarios, sensitive data escaped through tool inputs or logs while the final answer stayed clean.

The practical consequence is that the safest agent in the answer can still be the leakiest in the logs. If your instrumentation stores intermediate reasoning, treat it as a disclosure surface and decide on purpose whether to keep it. If you keep it, give it the same redaction, access control and retention as any other record of personal data.

Decide who reads, and for how long

  • Readers. List the roles that can query the logging backend. Debug access for the engineers on call is a different permission from analytics access for a product team.
  • Retention. Set an expiry per field class. Operational fields can live for months. Payloads, if you must keep them, should expire in days.
  • Alerting. Check what an alert carries. A page that quotes the failing request copies the payload into a chat tool.
  • Deletion. A deletion request has to reach the telemetry store, so record which stores hold personal data and how a record is found.

Use traces without overstating their coverage

AgentLeak can ingest OpenTelemetry and OpenInference spans. For application logs outside those spans, add explicit log events to the test trace. Compare the normalized events with the captured exporter output before interpreting the findings. Keep raw synthetic fixtures separate from production telemetry, and retain the settings and adapter version with each report.

Questions

Should I enable full prompt capture in production?

Make that an explicit data-handling decision. Check which fields your instrumentation captures, where they go, who can read them and how long they remain. Use synthetic fixtures to test those settings before sending production content.

Does hashing an email make it anonymous?

A stable unsalted hash can still be linked across events and guessed from candidate addresses. Prefer a purpose-specific identifier when correlation is needed, and treat linkable identifiers as sensitive metadata.

Can AgentLeak recover content missing from OpenTelemetry spans?

No. An adapter can normalize available content; it cannot reconstruct fields that were never captured. Test the logging boundary with synthetic data and document which channels the audit actually observed.

Sources

  1. OWASP Logging Cheat Sheet: data to exclude and log protection
  2. OpenTelemetry: handling sensitive data
  3. OpenTelemetry GenAI attribute registry
  4. AgentLeak OpenTelemetry integration, version 0.20.1

Run it on your own agent.

AgentLeak is open source and runs locally. The first scenario takes one command and no key.