The answer is clean. What about the file your agent wrote?
CSV exports, Markdown reports and temporary files are disclosure channels too. Test the artifact's contents, destination and failure paths before sharing it.
An analyst asks an agent for a count of open support cases. The reply says “There are twelve.” The attached CSV contains the underlying customer notes because the export helper serialized the complete records. The answer fulfilled the request, while the file introduced a separate disclosure.
Treat the artifact as its own output
Start with the request's contract. A count needs an aggregate; a case assignment report may need selected identifiers; a customer export may require a broader set of fields. Specify the fields the intended reader should receive, then check the file independently of the message announcing it.
List the intermediate artifacts too: scratch CSVs, cached tables, notebook outputs and files left behind after an exception. A final formatter can remove a column while an earlier copy remains in a shared workspace. Test the path that writes that first copy.
Capture content, not only a filename
The AgentLeak trace format includes a generated_file channel. Supply the text or structured content written to the artifact, with its source and destination. A trace saying only report.csv lets a detector inspect that path string, not the bytes stored there.
This synthetic example simulates a CSV export in memory. Install agentleak==0.20.1 and run it as Python. It checks that the registered marker appears in a finding on the file channel:
from agentleak import AgentLeakRunner, Trace
from agentleak.core.canary import CanarySet
marker = "CANARY_EXPORT_NOTE_7Q9"
trace = Trace(run_id="export-control", agent_name="reporter")
trace.add_event(channel="tool_response", source="support-db",
target="reporter", content={"private_note": marker})
trace.add_event(channel="generated_file", source="reporter",
target="shared-report.csv",
content=f"status,private_note\nopen,{marker}\n")
trace.add_event(channel="final_output", content="Report ready.")
report = AgentLeakRunner().analyze(
trace, canary_set=CanarySet(obvious=[marker]))
assert any(f.data_type == "canary" and f.channel == "generated_file"
for f in report.findings), "The file disclosure was missed"
print("File disclosure detected")This tests event analysis, not automatic filesystem scanning or an export application's access controls. For an actual file, capture the serialized content at the writer or extract it with an appropriate parser in the test harness. Confirm the extracted content reaches the trace before accepting a passing result.
Build an artifact inspection matrix
| Artifact | What the test should inspect | Separate delivery check |
|---|---|---|
| CSV or JSON | All exported fields, including nested values | Download permission and destination |
| Spreadsheet | Relevant sheets, cells and comments, including hidden content | Workbook sharing and temporary copies |
| Document or PDF | Extracted text and supported metadata; images require separate inspection | Attachment recipient and link access |
| Scratch output | Content written before success, retry or failure | Workspace permissions and cleanup |
The matrix is a proposed test plan, not a claim that one extractor covers every format. Record unsupported areas. If a parser fails or produces no text from a document that should contain it, report the inspection as incomplete.
Allow a field in one place and forbid it in another
The study behind AgentLeak sets the allowed fields per channel, not once for the whole run. A scheduling agent may need a patient’s name in a message to a billing agent and still must not write it into a file. Artifacts and logs are treated as restricted surfaces even when an agent transiently needs the data elsewhere, because a file outlives the run that wrote it and is read by people and systems the run never met.
| Field | Message to a specialist agent | Shared export file |
|---|---|---|
| Case status | Allowed | Allowed |
| Customer name | Allowed for the case it belongs to | Only for the report that is about that customer |
| Internal note | Only if the specialist needs it | Not allowed |
The table is an illustration of the shape of the contract, not a rule for your data. What matters is that each column is decided separately and written down. A single list of permitted fields hides the question the file raises, which is who will open it.
Plan for how long the file lives
A report that was correct to create can become a problem by staying. Give each artifact an owner and an expiry, and test the cleanup the way you test the content: run the task, let it finish and fail, then check what remains in the workspace. Scratch files left by a crashed export are the usual source of surprises, since nothing ever reached the step that would have deleted them.
Fix the export schema and check the recipient
Construct the output from an explicit field list, then serialize it. This makes an added database column less likely to appear in an export by accident. Re-run the same task and verify both that the forbidden field is gone and that the requested report remains useful.
Finally, test who can fetch the artifact. A report containing exactly the permitted fields can still reach the wrong account through a shared link. Content findings identify exposed values; authorization tests establish the permitted reader. Keep both results with the release evidence, including cleanup behavior on failure.
Questions
Does AgentLeak automatically open every path named in a trace?
No. The example analyzes content supplied in generated_file events. A path without the file's text does not establish coverage of its contents. Your capture or extraction step must provide the material you intend to inspect.
Is a file with personal data always forbidden?
No. An authorized export can legitimately contain personal data. State the permitted fields, recipient and purpose, then evaluate both the content and the delivery path against that contract.
Does text extraction cover images and hidden document content?
Only if the extraction process includes them. State whether the test inspected visible text, sheets, comments, metadata or OCR output. An empty extraction should be treated as incomplete coverage.
Sources
Run it on your own agent.
AgentLeak is open source and runs locally. The first scenario takes one command and no key.